google

Robots.txt is a text file placed in the root directory of a website to instruct web crawlers or search engine robots (also known as "bots") on how to interact with and ind