Go back to fronty page View most popular entries View latest additions Submit tutorials to UnixTutorials.info
UnixTutorials logo

Linux

How to Create and Configure robot.txt for Apache web server

Post date: February 17, 2009, 08:02 Category: Views: 4314 Comments
Tutorial quote: "Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

Comments

Web-based applications and online marketing solutions - LumoLink