Beware of spider traps!
Search engines, like Google, Yahoo! and Microsoft’s Bing, are the way the vast majority of internet users find the products, services and information they’re looking for online.
If you want people to find your little corner of the web, the last thing you want is to make it difficult for search engines to index the content on your website.
“Spider traps” make it harder for search engine spiders to “crawl” your website effectively
Search engines use automated software (known as a spider or a “bot”) to “crawl” the web. These spiders visit websites and follow every link they find, collecting information about each page.
Internet spiders thrive on a diet of text, text and more text. They don’t see the graphics, clever animations and other flashy bells and whistles that web designers routinely use to make sites look pretty.
Using visual elements is, of course, crucial to making your site attractive to human visitors when they arrive, but over-reliance on some of these features can stop some spiders in their tracks, preventing them from indexing your pages at all.
Make sure your site works for both search engines and your human visitors by making sure each page includes relevant text-based content; avoid flash-only websites and frames, which are difficult for spiders to crawl effectively; and make sure that every page on your site can be reached via a simple text-based hyperlink.
Common search-engine unfriendly web features
| Web Site Feature | Why it’s bad for your search visibility | Web Site Feature | Why it’s bad for your search visibility |
|---|---|---|---|
|
All flash website
|
Difficult for spiders to crawl – many will simply refuse to index all flash sites |
JavaScript navigation
|
Unless you implement a workaround for the spiders, they may have trouble following script-based navigation to other pages on your site. Make sure you have a normal, text-only link to every page on your site. |
|
Frames
|
Frames are notoriously difficult to implement effectively from a user experience perspective, are really necessary in very few instances, and often cause indexing problems for search engine spiders. |
Image Maps and other non-text navigation
|
Some spiders may have problems following these links. If you use image maps for navigation on your pages, make sure you have alternative text-based hyperlinks leading to the same pages. |
|
Dynamically generated pages and long URLs
|
Less of a problem than it used to be, but some spiders can have trouble with dynamically generated URLs that contain too many parameters (?W=XYZ). Try and configure your site to use “Search Engine Friendly” URLs, or at least restrict URL parameters to a maximum of three. |
AJAX generated text
|
See note for all flash sites above. |