Recently, Google had officially declared that GoogleBot will stop comply with a Robots.txt directive for the purpose of indexing. Distributors depending on the robots.txt noindex directive are valid up to September 1, 2019 to drop it and start utilizing other options.