Guruhowtos

The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed.

This tool provides results only for Google user-agents (such as Googlebot). Other bots may not interpret the robots.txt file in the same way. For instance, Googlebot supports an extended definition of the standard. It understands Allow: directives, as well as some pattern matching. So while the tool shows lines that include these extensions as understood, remember that this applies only to Googlebot and not necessarily to other bots that may crawl your site.

If a robots.txt file exists in the root directory of the host, this tool lists the information that Google has about it, including:

• A link to the current robots.txt file on your site.
• When Google last downloaded the file – if you’ve made changes to the file after this date and time, our cached version won’t reflect the changes
• The status of the file – the HTTP response we received when we tried to downloaded it. (Learn more about status codes.) If we receive a 404 (File Not Found) error, this doesn’t present a problem. You don’t have to have a robots.txt file – but if you don’t, bots will be able to crawl all the pages on your site.
• The MIME type – if the file is a type other than text, we can’t process it.
• Whether the robots.txt is blocking access to your home page or to any Sitemaps you’ve submitted.
• If we had trouble parsing lines in the file.