What is the limit of a robot.txt file?

[pihu147741]

Hello friends,

What is the limit of a robot.txt file?

 

[Sleekinfosolutions]

We can use one robots.txt file for a single website. You can set all instruction for robots within a robots.txt file.

 

[chiragm984]

Your robots.txt file must be smaller than 500KB. John Mueller of Google, reminded webmasters via Google+ that Google has a limit of only being able to process up to 500kb of your robots.txt file.

 

[arindamdutta16]

Robots is a catch all term for programs and automated scripts that crawl through the web and collect data from websites and anything else on the Internet that they can find. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

 

[Preeju M S]

Yes, there is a limit to robots.txt file in terms of filesize 500 KB. A maximum file size may be enforced per crawler.

 

[Ridhudhadwal]

robots.txt file for a single website

 

[exhibitionsconcept28]

use one robot.txt file in one website.

 

[auraiyaitsolutions]

We can use max 400-500kb size file for the website.

 

[riyaz1539]

robots.txt can be used for one website only and its size should not be greater thans 500kb