What is robot.txt file?

What is robot.txt file?

Hi,

What is robot.txt file?

Thanks

View Answers

January 26, 2011 at 12:36 AM

Hi,

The robot.txt file is places in the root of the website.

It allows the web masters to instruct the website pages for indexing in the search engine.

You can also disallow the search engine for indexing or crawling certain url's of your website.

Here is example of robot.txt file:

User-agent: *
Allow: /

Thanks

Ads









Related Tutorials/Questions & Answers:
What is robot.txt file?
ModuleNotFoundError: No module named 'filee'
Advertisements
ModuleNotFoundError: No module named 'filem'
ModuleNotFoundError: No module named 'filez'
ModuleNotFoundError: No module named 'filex'
ModuleNotFoundError: No module named 'filex'
what are the codes for uploading and downloading files in and from database?
ModuleNotFoundError: No module named 'filer'
files
ModuleNotFoundError: No module named 'cmsplugin-filer'
ModuleNotFoundError: No module named 'django-filer'
ModuleNotFoundError: No module named 'flask-filer'
ModuleNotFoundError: No module named 'flask-filer'
ModuleNotFoundError: No module named 'fluentcms-filer'
ModuleNotFoundError: No module named 'flask-filer'
ModuleNotFoundError: No module named 'flask-filer'
ModuleNotFoundError: No module named 'fluentcms-filer'
what are the jar files should i add to resolve the Exception(Class org.apache.jsp.index_jsp Not Found) ?
ModuleNotFoundError: No module named 'appomatic_django_filer'
ModuleNotFoundError: No module named 'cmsplugin_filer_album'
ModuleNotFoundError: No module named 'cmsplugin-filer-gallery'
ModuleNotFoundError: No module named 'cmsplugin-filer-image-translated'
ModuleNotFoundError: No module named 'django-ckeditor-filebrowser-filer'
ModuleNotFoundError: No module named 'djangocms-filer-video'
ModuleNotFoundError: No module named 'djangocms-filer-video'
ModuleNotFoundError: No module named 'django-cratis-filer'
ModuleNotFoundError: No module named 'django-filer-image'
ModuleNotFoundError: No module named 'django-filer-mod'
ModuleNotFoundError: No module named 'ripiu.cmsplugin-filer-svg'
ModuleNotFoundError: No module named 'ripiu.cmsplugin-filer-video'
ModuleNotFoundError: No module named 'appomatic_django_filer'

Ads