Guruhowtos

The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed.
This tool provides results only for Google user-agents (such [...]

The easiest way to create a robots.txt file is to use the Generate robots.txt tool in Webmaster Tools. Once you’ve created the file, you can use the Analyze robots.txt tool to make sure that it’s behaving as you expect.
Once you’ve created your robots.txt file, save it to the root of your domain [...]

A 404 page is what a user sees when they try to reach a non-existent page on your site (because they’ve clicked on a broken link, the page has been deleted, or they’ve mistyped a URL). A 404 page is called that because in response to a request for a missing [...]

A robots.txt file can make a huge impact on your WordPress blogs traffic and search engine rank. This is an SEO optimized WordPress robots.txt file. Keep in mind that if you mess up the robots.txt file by blocking too much, you could lose all of your rank.

No matter what the size of your website, you should include a detailed, text-based site map, with a link to every page and preferably, a short description of what each page offers. An excellent example of a site map can be found here. The advantage of using a site map is that you don\’t have to link to every page from your home page, but you should link to your site map from every page. Not only are site maps useful for visitors looking for specific information on your site, but they are great “spider food”, meaning they allow search engines to easily find and index every page on your site.

Most search engines index the content of title tags and consider it to be one of the most important factors in their relevancy algorithm. What you place in your title can make or break your ranking for particular search terms on the various engines. If you don’t include your most important search phrases within your title tag, you are overlooking a vital opportunity in your quest for higher search rankings

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.