Robots.txt
A file that tells crawlers which parts of a site they may access.
Robots.txt is a text file at the root of a site that gives crawlers instructions on which paths to crawl or skip. It manages crawl budget and keeps bots out of areas that should not be indexed. It is powerful and easy to misuse: a single stray rule can accidentally block an entire site from search, so it is worth checking carefully.