Guruhowtos

When browsing web files and directory, most bots, without consideration to .htaccess restrictions, have the capability to crawl and sniff files and folders located under your web root folder, even if these files and folders were not visually displayed from your web pages.

Take for an example, you displayed a file under http://domain.com/folder1/folder2/folder3/page.html

Most bots would initially crawled for targetted page file http://domain.com/folder1/folder2/folder3/page.html and later on would schedule a secondary deep crawl for possible files under

http://domain.com/folder1/folder2/folder3/

http://domain.com/folder1/folder2/

http://domain.com/folder1/

You could actually see bots coming back and forth to your computer host and start crawling subfolders and subpages of your public root files.

There are a lot of approach on how to deny access to particular web pages from being accessed and viewed from web. This entry covers an approach on how to use Apache directives to deny access to web files or particular set of file extensions from being viewed and browse by web clients

Basic Requirements

a. Existing Apache Setup On Linux Platform

Here are the following steps

Steps One

For example, we want to deny browse access to all files ending in XYZ (*.XYZ) using Apache directives.

Backup and modify your /etc/httpd/conf/httpd.conf . Search and look for the line

AccessFileName .htaccess

<Files ~ “^\.ht”>
Order allow,deny
Deny from all
</Files>
Steps Two

To deny access to web clients from viewing all files with XYZ file extensions (*.ZYZ), simply append similar lines like so

<Files ~ “^\.xyz”>
Order allow,deny
Deny from all
</Files>

Modify xyz to suit your file extension needs.

To deny access to particular file, simply

<Files ~ “private-file.txt”>
Order allow,deny
Deny from all
</Files>

Steps Three

Save and exit /etc/httpd/conf/httpd.conf . Restart HTTPD daemon service

# service httpd restart

Test your new Apache setup

# elinks http://domain.com/folder1/folder2/folder3/page.xyz

# elinks http://your-domain.com/yourpage.XYZ

All done.