You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
What is the limit of a robot.txt file?
- Thread starter ruhikhan
- Start date
virginoilseom
New member
I think google has a limit only 500Kb of a robots.txt file
ecartproduct
New member
Google currently enforces a size limit of 500 kilobytes (KB).
chiragm984
New member
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Martin smith
New member
Google providing 500kb limit of robots.txt file..
EmilieOlivia
New member
Robots.txt Files Must Be Smaller Than 500KB. Jan 30, 2012 • 8:57 am | (11) by Barry Schwartz | Filed Under Google Search Engine Optimization. Google's John Mueller reminds webmasters on his Google+ page that Google has a limit of only being able to process up to 500KB of your robots.txt file.