What is the limit of a robot.txt file?

Your robots.txt file must be smaller than 500KB. John Mueller of Google, reminded webmasters via Google+ that Google has a limit of only being able to process up to 500kb of your robots.txt file.
 
Robots is a catch all term for programs and automated scripts that crawl through the web and collect data from websites and anything else on the Internet that they can find. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.
 
Back
Top