Bulk up Your Robot: Sitemap Autodiscovery Supported by Major Engines
It’s been a long time coming, and while to Google, Yahoo, and Ask.com, Sitemaps Autodiscovery is old news, to MSN it’s brand new.
About three months after the formation of Sitemaps.org, and three months after Google, Yahoo, and Ask.com began supporting Sitemaps Autodiscovery, MSN’s Live Search has finally caught up.
According to this short, yet clear Search Engine Roundtable post, some are a bit frustrated at how long we’ve waited for this.
But don’t hold your breath; according to this WebmasterWorld thread, it could still be a couple of months before the service is widespread. Interestingly, “msndude” who participated in this particular WMW thread, encouraged us to look for updates on the LiveSearch blog, yet the most recent post is from June 1, 2007.
The option of adding a path to a sitemap.xml file allows webmasters to indicate to Google, Yahoo, Ask, and Live what pages of their site they want the bots to crawl and index (without manually submitting URLs), thus giving more control to the webmaster.
In addition to the list of URLs, webmasters are free to also add a variety of parameters to their sitemap.xml to tell the bots how often the pages are updated, the level of priority, and the last modification date. Not all of the parameters are required, but they do provide the search engine with additional information that can be helpful for crawl frequency.
Anyway, to bulk up your robots.txt file for Google, Yahoo, Ask, and soon Live, add the following line to your robots.txt file: