AI Citation SEO

AI Crawler Controls for Search, Training, and Agents

A practical guide to separating search discovery, model-training controls, user-triggered fetchers, robots.txt, noindex, and WAF access.

NavSEO Editorial · Updated July 20, 2026 · Reviewed July 20, 2026 · intermediate

AI crawler control is not one switch. Website owners need to separate four different questions:

Can a search or answer system discover and index the page?
Can the page appear as a source in a generated answer?
Can crawled content be used for model training?
Can a user-triggered agent fetch or interact with the page?

The user agents, controls, and outcomes differ. A rule aimed at model training may have no effect on search visibility, while a crawl block can prevent a system from reading a noindex directive on the blocked page.

Provider control matrix

Provider token or agent	Primary documented role	Main control	Important limitation
`Googlebot`	Google Search crawling, including Search features such as AI Overviews and AI Mode.	`robots.txt`, index controls, snippet controls, and participating Search Console controls.	Blocking Googlebot affects Google Search discovery, not only generative features.
`Google-Extended`	Controls use of crawled content for future Gemini model training and grounding in specified Gemini and Vertex AI products.	`robots.txt` product token.	It does not affect inclusion or ranking in Google Search.
`OAI-SearchBot`	Search discovery and source inclusion in ChatGPT search features.	`robots.txt` plus network access to published OpenAI IP ranges.	A blocked page may still appear as a navigational title and link when OpenAI learns the URL elsewhere.
`GPTBot`	Crawling content that may be used to train OpenAI foundation models.	`robots.txt`.	This setting is independent of OAI-SearchBot search inclusion.
`ChatGPT-User`	User-triggered visits from ChatGPT and custom GPT actions.	Application access controls and, where applicable, authentication.	Because the request is user initiated, `robots.txt` rules may not apply. It does not control ChatGPT Search inclusion.
`Bingbot`	Bing search crawling and the index used by Bing experiences.	`robots.txt`, meta robots, HTTP headers, and supported snippet controls.	Citation reporting is aggregated and does not prove ranking, authority, or causation.
`PerplexityBot`	Perplexity search discovery and source links.	`robots.txt` plus network access to published Perplexity IP ranges.	Perplexity documents this bot as separate from foundation-model training.
`Perplexity-User`	User-triggered page retrieval for an answer.	Application access controls and network policy.	Perplexity says this user-triggered fetcher generally ignores `robots.txt`.

Provider documentation changes. Verify the current user-agent token, affected products, and published IP ranges before changing production controls.

Search inclusion and training are separate

OpenAI explicitly documents independent controls for OAI-SearchBot and GPTBot. A site can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for potential model training.

Google makes a similar distinction between Googlebot and Google-Extended. Googlebot controls affect Google Search, including its AI features. Google-Extended is a separate product token and does not affect Google Search inclusion or rankings.

Do not copy a generic “block all AI bots” file without deciding what outcome the business wants. It can remove useful search or citation eligibility along with training access.

Robots.txt, noindex, and snippet controls

These controls solve different problems:

Control	What it does	Common mistake
`robots.txt` disallow	Asks compliant automated crawlers not to request matching URLs.	Assuming it guarantees that a known URL cannot appear as a title or link.
`noindex`	Asks supported search systems not to index the fetched page.	Blocking the crawler in `robots.txt`, which may prevent it from reading the directive.
`nosnippet` or snippet limits	Restricts how supported systems show page content in previews or generated experiences.	Treating snippet control as removal from the index.
Authentication	Prevents unauthorized retrieval of private content.	Publishing confidential material and relying on crawler directives for security.

If content must remain private, protect it with authentication or remove it from public access. Crawler directives are not security controls.

OpenAI’s publisher guidance notes that a disallowed URL learned through another source may still appear as a title and link in ChatGPT Atlas. It recommends noindex when the page should not appear that way, while also noting that the crawler must be allowed to fetch the page to read the meta tag.

Bing supports page-wide and section-level snippet controls, including data-nosnippet for excluding selected visible sections from supported search snippets and AI-generated answers while keeping the page discoverable.

WAF and CDN access

A permissive robots.txt file does not guarantee access. CDNs, bot-management products, rate limits, JavaScript challenges, geo rules, and origin firewalls can still block a documented crawler.

For important providers:

Use the provider’s current published IP ranges.
Verify both the user agent and the request source where the provider documents a verification method.
Review response codes in CDN and origin logs.
Check whether HTML, images, scripts, and API responses required for the main content are accessible.
Avoid permanent allow rules based only on a spoofable user-agent string.

Agent accessibility is a separate layer

Search crawlability does not guarantee that an agent can use a form, menu, filter, or checkout flow.

OpenAI’s publisher and developer guidance says ChatGPT Atlas uses ARIA labels, roles, and states to interpret page structure and interactive elements. This makes established accessibility work relevant to agent compatibility:

Use semantic HTML controls.
Give inputs and buttons accessible names.
Expose expanded, selected, invalid, and disabled states accurately.
Keep critical actions usable without ambiguous icon-only controls.
Test narrow layouts and authenticated states where agents may operate.

ARIA does not improve authority or guarantee a citation. It helps supported agents understand interactive interfaces.

Audit workflow

Inventory crawler and agent rules in robots.txt, meta tags, HTTP headers, CDN rules, and application authentication.
Write down the intended outcome for search inclusion, answer inclusion, training, and user-triggered access.
Compare each rule with current provider documentation.
Test representative public pages and intentionally restricted pages.
Review logs for response codes, challenges, and repeated failures.
Record the date, provider documentation URL, decision owner, and reason.
Recheck high-volatility controls quarterly or after a provider announcement.

Useful source references

Use crawler controls to express a deliberate policy. Do not treat them as an AI visibility tactic in isolation: useful public content still needs clear authorship, reliable evidence, internal discovery, and third-party trust.