You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
What is the limit of a robot.txt file?
- Thread starter ruhikhan
- Start date
A robots.txt file consists of one or more rules. Each rule blocks (or or allows) access for a given crawler to a specified file path in that website. Explanation: The user agent named "Googlebot" crawler should not crawl the folder http://example.com/nogooglebot/ or any subdirectories.
exploroapp
New member
There is a limit to robots.txt file in terms of filesize 500 KB.
Sleekinfosolutions
New member
Hi,
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Local website marketing service India | Global Search Engine Optimization company India
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Local website marketing service India | Global Search Engine Optimization company India
exhibitionsconcept28
Active member
thanks for sharing.....
dombowkett
Active member
We can max. 500kb size for robots.txt file for fast results.