Search Engine Spiders

What are Spiders?

Spiders can be thought of as minion or informants. Web spiders are released onto a few popular sites and from there they follow links absorbing a copy of content in their path.

Spidering Schedule

The internet is starting to get so large that it is hard to spider the entire net more than about once or twice a month. For that reason, spiders are used to regularly update the sites and pages that are viewed as more important while the rest of the web is updated every month or two. Pages which are known to change frequently (such as blogs, forums, and news sites) are spidered frequently, sometimes multiple times a day. The spiders insert this data into a catalog for further processing.

Spider Roadblocks: Dynamic Programming & Java

The spiders are smart and dumb. They are very good at following text links and any old regular hyper link. These same spiders struggle to list complex dynamic pages (good at following simple dynamic pages) and struggle following java script links (Google has been testing a spider that spiders JavaScript).

Spider Precautions With Dynamic Content

If spiders tried to spider a site that issued session ID's or cookies the site can appear much larger than it is to the spider, the spiders can get stuck in an endless loop, or the spiders can run through dynamic content so fast that they crash the server. For that reason spiders are instructed not to follow deep dynamic links without taking pauses between page requests. You still can list dynamic pages easily though.

While search engines now list dynamic content they view everything the record as HTML and it is easy to think of a search engine spider as a HTML photographer.