Guruhowtos

Ever wondered why so many clients are interested in a file called robots.txt which you don’t have, and never did have?
These clients are called robots (also known as crawlers, spiders and other cute names) – special automated clients which wander around the web looking for interesting resources.
Most robots are used to generate some kind of web index which is then used [...]

Most linux servers I know and all servers I have been managing boots into runlevel 3 specially those unattended servers being managed remotely from far remote locations.
With that in mind, data file transfers are done via terminal commands between two or more hosts, locally from the network or from the internet. Here are two ways [...]

Rdiff-backup is similar to rsync linux command. Rdiff-backup uses librsync algorithm library file which is also used by rsync linux command,, but rdiff-backup never uses rsync to do its backup functions. Rdiff-backup is also similar to rsnapshots as rdiff-backup creates a backup copy, an actul mirror/clone of the data being back up to a separate remote location, the difference is rdiff-backup creates a statistical file, sessions, and meta data to identify changed data for further incremetal backup operation. Rdiff-backup makes use of ssh connection during the data transfer between source and destination backup host.

The robots.txt analysis tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn’t understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed.
This tool provides results only for Google user-agents (such [...]

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.