Bots

[Ack]

So I keep seeing at the bottom things like the MSN Bot, Yahoo Bot, Google Bot, and so on and so forth. Exactly what is it these guys do, anyway? Do they exist merely to transmit back to their bases what information is added or updated, do they monitor site traffic, or are they just there to annoy us? Seriously, I'm interested. Anybody know?

[lordofduct]

I've been wondering too

I think they are the bots and spiders that search the web and catalog websites for the search engine. Probably why I see them on more at the wee hours of the morning, then during the middle of the day.

[andymol21]

There's an MSN one on here now, and I've seen a couple of Google ones. Strange, I have no clue at all!

[disorderlyvision]

and why are they always listed under the registered users instead of guests

[Ziggy]

I, too, have wondered.

[disorderlyvision]

...because enquiring minds want to know

from wiki

A Googlebot is a search bot used by Google. It collects documents from the web to build a searchable index for the Google search engine.

If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file,[1] or by adding the meta tag <meta name="Googlebot" content="nofollow" /> to the webpage. [2] Googlebot requests to Web servers are discernible from their user-agent string 'Googlebot'.

Googlebot has two versions, deepbot and freshbot. Deepbot, the deep crawler, tries to follow every link on the web and download as many pages as it can to the Google indexers. It completes this process about once a month. Freshbot crawls the web looking for fresh content. It visits websites that change frequently, according to how frequently they change. Currently Googlebot only follows HREF links and SRC links.[verification needed]

Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from another known page on the web in order to be crawled and indexed.

A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Webmaster Tools" that allow website owners to throttle the crawl rate.

[Ack]

So what is the MSN Mediabot then? Is Keith Olbermann interested in retro gaming?