Big Data and Hadoop Training @$80 or Rs. 6,000 only - valid till Oct 17th Apply Now!

What is robot.txt file?

Ads
 

What is robot.txt file?

Hi,

What is robot.txt file?

Thanks

View Answers

January 26, 2011 at 12:36 AM

Hi,

The robot.txt file is places in the root of the website.

It allows the web masters to instruct the website pages for indexing in the search engine.

You can also disallow the search engine for indexing or crawling certain url's of your website.

Here is example of robot.txt file:

User-agent: *
Allow: /

Thanks

Ads









Related Tutorials/Questions & Answers:
What is robot.txt file?
What is the purpose of the following files having extensions: frm, myd, and myi? What these files contain?
Advertisements
what are the codes for uploading and downloading files in and from database?
files
files
what are the jar files should i add to resolve the Exception(Class org.apache.jsp.index_jsp Not Found) ?
Concatenate two pdf files
configuration files are used in Struts
autocomplete(): Spring mvc with jquery: I am not getting correct value in the text filed. Please help me
Merging multiple PDF files - Framework
Compare 2 files
header files
library files - Development process
Uploading files
validation files
files
files
files
files
files
Files
configuration files - Java Beginners
Reading files in Java
Concatenate pdf files using command argument
How Compare 2 xml files with JDOM - Java Beginners
What is CVS
files uploding and downloading codes
invoking exe files on sound
download pdf files
php merge files
replacing regex in large files

Ads