Blocking “AI bots” sounds like one switch. It is not. The same company may operate separate agents for model development, search indexing and a user’s direct request. A blanket rule can therefore stop uses you object to and discovery you wanted to keep.

The useful decision is not allow AI or block AI. It is: which automated use of which content creates enough value to justify access, under what conditions, and how will we verify the result?

This guide gives small publishers a repeatable way to answer that question. It does not determine copyright rights, licensing terms or crawler compliance in your jurisdiction. Those questions require appropriate legal advice and evidence beyond a robots.txt file.

Start with purpose, not bot names

Create three policy buckets before editing anything:

Purpose Typical publisher question Default starting position
AI search and answer discovery Could this surface our pages with citations or navigational links? Allow while measuring referrals and crawl cost
Model training or development Do we want our content collected for future model development? Make an explicit rights and business decision
User-directed retrieval Should an assistant fetch a page because a user asked for it? Allow unless security, licensing or server-load concerns outweigh the benefit

These are starting positions, not universal recommendations. A subscription research publisher may protect article bodies while allowing public abstracts. A product documentation site may prioritize assistant access because successful retrieval helps customers. A small affiliate publication may want AI-search discovery but decline model-training collection.

The crawler names are not interchangeable

Current official documentation shows why a single “block OpenAI” or “block Anthropic” rule is too coarse.

OpenAI

OpenAI documents separate controls:

  • OAI-SearchBot is used to surface sites in ChatGPT search results. OpenAI says sites opted out of it will not be shown in ChatGPT search answers, although navigational links may still appear.
  • GPTBot crawls content that may be used to improve and train generative AI foundation models. OpenAI says disallowing it indicates that site content should not be used for that training purpose.
  • ChatGPT-User can visit a page because a user or Custom GPT requested it. OpenAI says it is not an automatic web crawler and that robots.txt rules may not apply to these user-initiated actions.

The practical implication is important: a publisher can express allow search, decline training by treating OAI-SearchBot and GPTBot differently.

Anthropic

Anthropic also documents three roles:

  • ClaudeBot for model development;
  • Claude-SearchBot for improving search-result quality;
  • Claude-User for retrieval initiated by a Claude user.

Anthropic warns that disabling Claude-SearchBot may reduce visibility and accuracy in Claude search results, while disabling Claude-User can prevent retrieval in response to a user’s question. Its documentation says these bots honor industry-standard robots.txt directives.

Google

Google-Extended is a product token, not a separate HTTP crawler user agent. Google says it controls specified uses of crawled content for training future Gemini models and for grounding in Gemini products. Google also states that Google-Extended does not affect inclusion or ranking in Google Search.

Do not confuse Google-Extended with Googlebot. A publisher that wants ordinary Google Search visibility should avoid copying a generic “block Google AI” snippet without checking exactly which token it targets.

Common Crawl

Common Crawl operates CCBot and makes its crawl dataset broadly available. Its official FAQ says CCBot honors robots.txt, supports Crawl-delay, and can be blocked with a CCBot-specific group. Blocking CCBot is therefore a decision about access to a public web dataset, not a direct switch for one named assistant or search product.

Build a crawler access register

Do not manage this policy as undocumented lines in a text file. Keep a small register beside your website configuration:

Field What to record
Operator and token Exact documented crawler or product token
Stated purpose Training, search, user retrieval, archiving or unknown
Current action Allow, monitor, throttle, block or enforce
Content scope Whole site or selected paths
Expected value Referral, citation, customer support, research exposure or none identified
Expected cost Bandwidth, licensing concern, paywall leakage or operational load
Evidence source Official documentation URL and checked date
Review date When the decision must be reconsidered
Owner Person responsible for changing and validating the rule

The “unknown” category matters. A user-agent string alone does not prove identity, purpose or compliance. Record uncertainty rather than assigning a familiar company name to traffic you have not verified.

Use a five-question decision test

Score each crawler separately. A yes/no worksheet is more defensible than an emotional blanket policy.

1. Is the purpose documented?

Find the operator’s current crawler documentation. Record the exact token, purpose, published IP verification method if available, and the difference between automatic and user-triggered access.

If you cannot establish who operates the crawler or why it accesses the site, start with monitoring or blocking at the enforcement layer—not with a permanent policy presented as fact.

2. Is there a plausible reader or business benefit?

Define the outcome before allowing access:

  • qualified referral visits;
  • cited answers that build recognition;
  • customers retrieving documentation inside an assistant;
  • lower support effort;
  • a licensing or paid-crawl arrangement.

“AI visibility” is not an outcome by itself. If no measurable benefit exists, write that down.

3. Is the content suitable for that use?

Separate public discovery material from content that should not be copied freely:

  • public articles and documentation;
  • licensed databases;
  • member-only or paywalled pages;
  • personal data;
  • draft or staging paths;
  • duplicate, parameterized or deprecated URLs.

Robots rules are not access control. Sensitive content must be protected through authentication and server-side authorization, not merely disallowed in robots.txt.

4. Can the decision be enforced and verified?

A robots.txt directive is a request to compliant crawlers. Cloudflare explicitly notes that compliance is voluntary. If the policy requires technical prevention, use an enforcement layer and verify crawler identity rather than trusting a self-declared user agent.

For Cloudflare-hosted sites, AI Crawl Control can show crawler activity, robots violations and per-crawler actions. Cloudflare says its free-plan identification relies on user-agent strings, while stronger detection is available with Bot Management. That limitation belongs in the decision record.

5. What would make you reverse the decision?

Set the rollback rule before deployment. Examples:

  • allow a search crawler if it produces five qualified referrals in 30 days;
  • block a training crawler if there is no agreement or strategic reason to permit collection;
  • throttle a crawler if it exceeds a defined request rate;
  • reopen access after the operator publishes clearer purpose and verification documentation.

Without a review trigger, a temporary experiment becomes an invisible permanent policy.

Three policy patterns

Use these as patterns, not copy-and-paste legal instructions. Confirm every token against current operator documentation.

Pattern A: allow search, decline selected training crawlers

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This separates OpenAI search discovery from GPTBot training use and applies Google’s product token without blocking Googlebot. It says nothing about Anthropic or Common Crawl until you add explicit decisions for them.

Pattern B: allow public resources, protect a licensed section

User-agent: ClaudeBot
Disallow: /members/
Allow: /

User-agent: GPTBot
Disallow: /members/
Allow: /

This may reduce compliant crawling of the specified path, but it does not secure /members/. The server must still require authentication.

Pattern C: reduce crawl rate where supported

User-agent: CCBot
Crawl-delay: 2

Common Crawl and Anthropic document support for Crawl-delay; not every crawler does. Do not assume one extension works globally.

Robots.txt and enforcement solve different problems

Treat policy expression and technical enforcement as two layers:

Layer What it does What it cannot prove
robots.txt Publishes path-level preferences for identified agents That every requester will comply
Verified crawler data Helps confirm known operators The purpose of unidentified traffic
WAF or AI crawler control Technically allows or blocks matched traffic That the business policy is correct
Authentication Protects non-public content How public content may be reused after access
Content signals or licensing terms Express intended uses or commercial conditions Universal acceptance or enforcement

Cloudflare’s managed robots.txt can add Content Signals describing preferences for search, AI input and AI training. Cloudflare presents those signals as an expression of content-use preferences and notes that Google Search Console may report unfamiliar directives without observed impact on crawling. This is an evolving mechanism, not a replacement for operator-specific documentation or legal advice.

Run a 30-day controlled change

Change one policy group at a time. Record the exact deployment time and retain the previous file.

Before the change

Capture a 14- or 30-day baseline:

  • requests and bandwidth by verified crawler;
  • top requested paths;
  • HTTP status distribution;
  • server load attributable to bots;
  • referral sessions from AI services;
  • conversions or engaged visits from those referrals;
  • known citations or navigational appearances.

During the change

Check after 24 hours, seven days and 30 days. OpenAI notes that search systems may take about 24 hours to adjust after a robots.txt update, so an immediate check is not a meaningful final result.

Use a log like this:

Date Token Policy change Requests Bandwidth Referrals Qualified outcomes Violations Decision
10 Aug OAI-SearchBot Allow / — — — — — Baseline starts
17 Aug OAI-SearchBot No change Continue / revert

Blank cells are deliberate: they must be filled from your own logs and analytics. They are not BenPicks results.

After 30 days

Classify the outcome:

  • Keep — measurable value and acceptable cost;
  • Narrow — value exists, but only for selected paths;
  • Throttle — value exists, but crawl load is excessive;
  • Block — cost or rights concerns outweigh the measured value;
  • Unresolved — evidence is insufficient, so extend the test rather than inventing a conclusion.

Common mistakes

Blocking the company instead of the purpose

One operator may publish multiple controls. Blocking every token can suppress search or user-directed retrieval when the actual objection concerned model training.

Treating robots.txt as security

Disallowing a private path does not make it private. Protect non-public content at the origin.

Copying a stale bot list

Crawler names, purposes and verification methods change. A long list copied from a forum becomes wrong silently. Maintain a short, documented register with review dates.

Measuring only crawl volume

Fewer requests can save bandwidth while reducing citations or useful retrieval. Pair cost metrics with referral and outcome metrics.

Claiming that a rule controls training already completed

A crawl directive governs future access by compliant agents. It does not demonstrate that historical copies were removed or that downstream datasets changed.

A practical starting policy for a small publisher

For a publisher that depends on discovery but has no training license, a defensible starting point is:

  1. keep ordinary search crawlers allowed;
  2. distinguish AI-search tokens from training tokens;
  3. allow documented search and user-retrieval agents while measuring value;
  4. make an explicit per-operator decision for model-training crawlers;
  5. protect private content with authentication;
  6. use enforcement only where a voluntary directive is insufficient;
  7. review the register every quarter or after a material documentation change.

This is not automatically the right policy for BenPicks or for every publisher. It is a process that makes the trade-off visible and reversible.

Limitations

Crawler documentation and product behavior can change. A user-agent can be spoofed, and the absence of a visible crawler request does not prove that content is absent from every index or dataset. Referral data also undercounts influence when an AI answer produces no click.

This article does not assess contractual rights, copyright exceptions, database rights or the legal effect of Cloudflare Content Signals. It also does not claim that BenPicks has run the 30-day experiment or measured referral effects from the agents discussed.

Bottom line

Do not begin with a blacklist. Begin with a register of purposes, content classes, expected value and enforcement needs. Separate training from search and user-directed retrieval; deploy one reversible change; then keep, narrow, throttle or block based on evidence.

The strongest policy is not the longest robots.txt. It is the one your team can explain, measure and update without accidentally removing a discovery channel it intended to preserve.

Sources