How to Allow AI Citation Without Allowing Model Training

Yes, you can allow AI citation while blocking model training, because the major AI engines run separate crawlers for each job and expose separate robots.txt controls. Allow the search crawlers, OAI-SearchBot, PerplexityBot, and Googlebot, so AI answers can still surface and cite your pages, and disallow the training controls, GPTBot and the Google-Extended token, to opt out of OpenAI and Gemini model training. Setting the two decisions independently in one robots.txt file keeps you citable without feeding the models.

Here is what to allow or disallow, and why, for each crawler and control:

Why Citation and Training Are Separate Decisions

AI citation and model training are two separate decisions because the engines that power AI answers use different crawlers for each. Citation depends on the search crawlers that read and surface your pages, while training depends on a different set of crawlers and tokens that feed model development. Because the jobs are split, one robots.txt file can allow citation and disallow training at the same time.

Engines Run Different Crawlers for Different Jobs

Each AI engine splits its crawling across named user-agents, so you control them one at a time. OpenAI uses OAI-SearchBot for ChatGPT search citations and GPTBot for model training, and the two settings are independent of each other. Perplexity uses PerplexityBot to crawl the pages it can cite, while Google uses Googlebot to index pages and the Google-Extended token to govern AI training.

Because the crawling jobs are split this cleanly, one robots.txt file can allow the citation crawlers and disallow the training crawlers without conflict. For a fuller picture of how AI search engines crawl sites differently across ChatGPT, Perplexity, and Google, the per-engine differences matter more than any single blanket rule.

The Costly Mistake Is Blocking Search to Stop Training

The expensive mistake is disallowing a search crawler in an attempt to stop training, which quietly removes a brand from AI answers. The page still loads normally for people, so the lost citation stays invisible until someone checks. The cost is real because a large share of buyers now begin in AI answers: 51% of business-to-business (B2B) software buyers start their research with an AI chatbot rather than a traditional search engine (G2).

In our work with B2B brands, an over-broad bot block is one of the most common reasons a page stops being cited. The fix is to target the training crawlers specifically, GPTBot and the Google-Extended token, rather than blocking all automated traffic.

Which Crawlers to Allow for AI Citation

AI citation requires keeping the search crawlers allowed, because those are the crawlers that read your pages and let AI answers surface and link them. The search crawlers that matter are OAI-SearchBot, PerplexityBot, and Googlebot. Allowing all three keeps you eligible to be cited across ChatGPT, Perplexity, and Google.

Allow the Search Crawlers

Allowing the search crawlers is what keeps each engine able to cite you. Allow OAI-SearchBot so your pages can appear and be cited in ChatGPT search answers. Allow PerplexityBot so Perplexity can surface and link your pages, and allow Googlebot so you stay indexed for Google Search and Google AI Overviews.

These three are allow decisions you make deliberately, not defaults you can assume. A broad security rule or a catch-all Disallow can remove any of them without touching your training settings, which is why the allow list is worth stating explicitly.

Remember Overviews Depend on Googlebot, Not Google-Extended

Google AI Overviews are built from the Googlebot-crawled index, not from a separate AI crawler. The Google-Extended token controls whether your content trains or grounds Gemini, and it does not affect Search indexing or ranking. So disallowing Google-Extended keeps you eligible for Overviews while opting your content out of Gemini training.

The distinction matters because Overviews reach an enormous audience: Google AI Overviews reach about 2 billion monthly users (TechCrunch). Blocking Googlebot to avoid Gemini training would forfeit that reach, when the correct, narrower control is the Google-Extended token.

Which Crawlers to Disallow for Model Training

Model training is controlled by a different, smaller set of robots.txt directives than citation. To opt out, you disallow GPTBot and the Google-Extended token, the two controls the major engines expose for training. Because search runs on separate crawlers, these disallow rules do not remove you from AI answers.

Disallow the Training Controls

Disallowing the training controls is a narrow, deliberate action. Disallow GPTBot to opt your content out of training OpenAI's generative models, and disallow the Google-Extended token to opt out of Gemini training and grounding. Neither setting removes you from search, because ChatGPT search and Google Search read your pages through OAI-SearchBot and Googlebot instead.

That separation is the whole point: the training disallows and the citation allows live in the same file and do not interfere. If you want the mechanics of how llms.txt and robots.txt differ for AI optimization, the controls described here all sit in robots.txt, which every major AI crawler reads.

Decide User-Initiated Fetches Separately

User-initiated fetches are a separate policy question from automatic crawling. Agents such as ChatGPT-User and Perplexity-User fetch a page because a person asked a question about it, and they may not follow robots.txt crawl rules the way automated crawlers do. Treat them as their own decision rather than folding them into your training or citation rules.

Document which user-initiated agents you allow, so the overall access policy stays auditable and a later reviewer can see the reasoning. This matters most when a security team tightens bot rules and needs to know which agents were allowed on purpose.

How to Set and Verify the robots.txt Rules

robots.txt is where all of these decisions are set, one User-agent block at a time. The file lets you allow the search crawlers and disallow the training controls in the same place, then confirm the result in your logs. Implementation and verification are two steps, and skipping the second is how silent blocks survive.

Set Per-Crawler Rules in One File

Setting the rules means adding one User-agent block per crawler with the Allow or Disallow you intend. Keep your priority pages allowed for OAI-SearchBot, PerplexityBot, and Googlebot, and disallowed for GPTBot and the Google-Extended token.

Avoid a broad catch-all Disallow under User-agent: *, which sweeps in the search crawlers along with everything else. Teams we work with often reach for a single blanket rule, which is exactly how a citation crawler gets blocked by accident. If you manage the file through a content management system (CMS), the steps for configuring AI crawlers on WordPress and other CMS platforms keep the per-crawler blocks intact.

Confirm Citation Crawlers Still Reach You

Confirming the citation crawlers still reach you is the step that proves the policy worked. After the change, check your server or content delivery network (CDN) logs for OAI-SearchBot, PerplexityBot, and Googlebot returning a 200 status on your priority pages, not a 403 or 429. Allow time for each engine to re-crawl and apply the new rules, since the change is not instantaneous.

VisibilityStack's Crawl Assurance Engine checks whether AI citation and training crawlers reach your priority pages and flags the blocks that silently remove you from AI answers. Re-verify after any CDN or security change, so a firewall rule does not block a crawler you meant to allow.