Updated July 2026 · 9 min read

AI Crawler Governance: How to Control AI Bot Access to Your Site

TL;DR

AI providers publish different user agents for search, training, and other product functions. Choose access deliberately using each provider's current documentation, then confirm the resulting status codes in server logs. Allowing a bot does not guarantee inclusion or citation. The proposed llms.txt file is not an access control, and Google Search says it ignores the file.

Why AI Bot Governance Matters

Provider documentation now distinguishes model-development crawlers, search crawlers, and user-initiated fetchers. A single blanket rule can therefore allow or restrict uses that a publisher intended to treat differently.

Robots.txt is only one layer. CDN, firewall, authentication, rendering, and origin behavior can produce a different result from the published rule. Access is also only an eligibility condition: a successful fetch does not guarantee that a system will index, quote, link to, or recommend the page.

Common Provider Controls to Review

Token or agent	Operator	Documented purpose	Decision
GPTBot	OpenAI	Potential model improvement	Publisher policy
OAI-SearchBot	OpenAI	Surfacing sites in ChatGPT search	Publisher policy
ChatGPT-User	OpenAI	User-initiated page visits	Review separately
ClaudeBot	Anthropic	Potential model training	Publisher policy
Claude-SearchBot / Claude-User	Anthropic	Search indexing / user-directed retrieval	Review separately
PerplexityBot / Perplexity-User	Perplexity	Search results / user-requested fetches	Review separately
Googlebot	Google	Google Search, including its AI features	Search policy
Google-Extended	Google	Control token for specified Gemini training and grounding uses; not a separate crawler	Publisher policy

Key distinction: Google documents Googlebot, not Google-Extended, as the robots.txt control for Search and its AI features. Google-Extended has no effect on inclusion or ranking in Google Search. Provider names and purposes can change, so verify them in current first-party documentation before deployment.

Configuring robots.txt for AI Bots

The following is an illustrative policy, not a universal recommendation. Replace each choice after legal, product, licensing, and infrastructure review; use the provider's exact current token:

# Traditional search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Example search and user-fetch policy
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Example model-development policy
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Block admin/private areas for all bots
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/

CDN and WAF Bot Blocking

A CDN or Web Application Firewall can override the practical effect of robots.txt by challenging or blocking requests. Inspect the configuration and logs instead of assuming a named security mode treats every provider in a particular way.

✓ Identity: Match the user agent with provider-published IP ranges or verification guidance; a user-agent string alone is easy to spoof.
✓ Rules: Compare CDN and application-layer decisions with the policy documented in robots.txt.
✓ Rate limiting: Base limits on observed request volume, cache behavior, and origin capacity.
✓ Responses: Check status, body, redirect chain, and challenge pages for verified requests.

Rendering Architecture and Accessible HTML

Crawler rendering capabilities differ and are not fully documented for every product. Returning essential text, links, titles, and metadata in the initial HTML avoids depending on JavaScript execution and also supports conventional search crawlers and accessibility tools.

Architecture	AI Visibility	Why
Server-Side Rendering (SSR)	Broadly accessible	Essential content can be present in the initial response
Static Site Generation (SSG)	Broadly accessible	Pre-built HTML can contain the complete page
Incremental Static Regeneration (ISR)	Broadly accessible	Static HTML can be refreshed on the site's schedule
Client-Side Rendering (CSR)	Requires testing	Essential content may depend on script execution

For example, inspect whether pricing, product specifications, and expandable answers are present in the response body before interaction. If they are absent, a fetcher that does not render that interface will not receive them.

Next.js, Nuxt, Astro, and other frameworks offer server or static rendering options. A full migration is not always necessary: first identify the important routes and data that are missing from initial HTML, then choose the smallest reliable rendering change.

The Complete AI Bot Governance Checklist

□ Each provider token has an explicit, documented allow or disallow decision
□ CDN/WAF behavior matches that policy for verified requests
□ Essential content and links are present in accessible HTML
□ Pricing, specifications, and key answers are tested without interaction
□ Any llms.txt experiment names the supporting application and is verified in logs
□ Server logs verified — AI bots receiving 200 status codes
□ No CAPTCHA or challenge pages served to AI bots
□ Policy owner and review date are recorded because provider controls can change

Primary References

Continue Learning

Frequently Asked Questions

AI bot governance is the documented process for deciding which automated agents may fetch which public URLs, implementing those choices in robots.txt and infrastructure, and checking the result in server logs. Each provider and user agent must be assessed separately.

That is a publishing-policy decision. OpenAI documents GPTBot for potential model improvement, while OAI-SearchBot is the separate agent used to surface sites in ChatGPT search. A GPTBot rule should not be presented as a ChatGPT search control.

OpenAI documents GPTBot for potential model training and OAI-SearchBot for surfacing sites in ChatGPT search. Review the current first-party documentation before setting policy. Blocking one agent can remove that documented access path but does not prove how every answer or citation is produced.

It can. Rendering capabilities differ by crawler and may change. Putting essential text and links in the initial HTML is the most broadly accessible approach; test the actual response and provider rather than assuming every bot executes, or never executes, JavaScript.

Check your robots.txt for explicit blocks on GPTBot, ClaudeBot, or OAI-SearchBot. Review your CDN/WAF settings (Cloudflare, Sucuri) for bot-blocking rules. Use server logs to verify AI bots are successfully crawling your pages and receiving 200 status codes.

Request volume varies by site and provider. Measure verified requests, response sizes, error rates, cache behavior, and origin load in your own logs, then apply proportionate rate limits. Do not assume either negligible load or a visibility benefit.

Ready to Scale Your SEO?

Generate optimized content, review it with SEO checks, and publish to WordPress from one workflow.

Start 3-Day Free Trial