Which is the best programming language to write a web bot?

Question

I want to know which programming language provides a good number of libraries to program a web bot? Something like crawling a web page for data. Say I want to fetch weather for weather.yahoo.com website.

Also, will the answer be the same for an AI desktop bot?

Answer 1

It depends on where your Bot would work or “live”, and what it would have to do. I would like to ask what level are you working at, meaning how fast does it need to be, what type of payload and how many will it need to parse.

Python would be fine for small projects or if you had the time to wait for it to go through a lot of data.

The dilemma with Python, Ruby, Php, or even Javascript, is that while they can be relatively fast these days, they are still slower than a well designed compiled language.

(Before someone states Java is not compiled, it is, just not to machine language), in almost every case well-designed java code should run faster than well-designed python code, and faster than ruby.

So, let us call it two levels: 

1. If you are looking at massive data and speed needs, go with C or C++.

2. If you need somewhat mid-level speed or larger data, I would say C# or Java (Or Python using Jython).

Here is how you could implement it in Python:

from urllib2 import urlopen

from BeautifulSoup import BeautifulSoup

soup=BeautifulSoup(urlopen("http://weather.yahoo.com/").read())

for x in soup.find(attrs={"id":"myLocContainer"}).findAll("li"):

  print x.a["title"], x.em.contents

Prints:

Full forecast for Chicago, Illinois, United States (Haze) [u'35...47 °F']

Full forecast for London, Greater London, England (Light Rain) [u'43...45 °F']

Full forecast for New York, New York, United States (Partly Cloudy) [u'42...62 °F']

Full forecast for San Francisco, California, United States (Partly Cloudy) [u'51...70 °F']