Robots.txt

Robots.txt manages crawler access. It can reduce crawl waste, protect low-value paths from crawling, and point crawlers toward XML sitemaps.

It does not guarantee that a URL will stay out of search results. For index control, use the right page-level or response-level signal, such as noindex, removal, redirects, canonicalization, or an appropriate status code.

What robots.txt can do

• Allow or disallow crawler access to URL paths.
• Reference XML sitemap locations.
• Reduce crawling of low-value paths such as internal search or parameter combinations.
• Help prevent crawlers from spending time on duplicate or infinite URL spaces.

Practical checks

• Important pages are not blocked.
• Resources needed for rendering are not accidentally blocked.
• Rules are tested before launch or migration.
• Sitemap references point to live sitemap files.
• Disallowed paths are intentional and documented.
• Crawl blocks are not used where noindex or redirects would be more appropriate.

Common mistakes

Do not block a page in robots.txt and expect Google to read a noindex tag on that blocked page. If the crawler cannot fetch the page, it may not see the directive.

Robots.txt is strongest when it supports a clear crawl strategy: help crawlers spend time on important pages and avoid low-value URL traps.