Sep 5, 2017
0 0

What is Robots.txt?

Written by

Robots.text can be defined as a text-based file that is put on websites when site owners do not want the search bots to visit certain pages because they could contain classified information or they may only want their browser versions indexed. It is the norm for search engines to visit websites and add the pages to their indexes.

The robots.text file should be in your main directory because the agents first search this location. They do not browse the entire site. If they fail to locate the robots.text file in the main directory, they index all the pages.

Spread the love
Article Tags:
Article Categories:

Leave a Comment

Your email address will not be published. Required fields are marked *