# How to Allow AI Citation Without Allowing Model Training Yes, you can allow AI citation while blocking model training, because the major AI engines run separate crawlers for each job and expose separate robots.txt controls. Allow the search crawlers, OAI-SearchBot, PerplexityBot, and Googlebot, so AI answers can still surface and cite your pages, and disallow the training controls, GPTBot and the Google-Extended token, to opt out of OpenAI and Gemini model training. Setting the two decisions independently in one robots.txt file keeps you citable without feeding the models. **TL;DR** - Yes, you can allow AI citation while blocking model training, because the major engines use separate crawlers for each purpose. - Allow the citation crawlers, OAI-SearchBot, PerplexityBot, and Googlebot, so AI answers can still surface and cite you. - Disallow the training controls, GPTBot and the Google-Extended robots.txt token, to opt your content out of model training. - robots.txt controls these per crawler, so you set the two decisions independently in one file. - Blocking a search crawler to stop training is the common mistake that quietly removes a brand from AI answers. - After changing robots.txt, verify in your logs that citation crawlers still return a 200 on your priority pages. Here is what to allow or disallow, and why, for each crawler and control: | Crawler or Control | Decision | What It Controls | |---|---|---| | OAI-SearchBot | Allow | Appear and be cited in ChatGPT search answers | | GPTBot | Disallow | Opt content out of training OpenAI's generative models | | PerplexityBot | Allow | Be surfaced and linked in Perplexity answers | | Googlebot | Allow | Stay indexed for Google Search and Google AI Overviews | | Google-Extended | Disallow | Opt out of Gemini training and grounding; no effect on Search indexing | | robots.txt | Per-user-agent | Rules are set per crawler, so citation and training are independent decisions | ## Why Citation and Training Are Separate Decisions AI citation and model training are two separate decisions because the engines that power AI answers use different crawlers for each. Citation depends on the search crawlers that read and surface your pages, while training depends on a different set of crawlers and tokens that feed model development. Because the jobs are split, one robots.txt file can allow citation and disallow training at the same time. ### Engines Run Different Crawlers for Different Jobs Each AI engine splits its crawling across named user-agents, so you control them one at a time. OpenAI uses OAI-SearchBot for ChatGPT search citations and GPTBot for model training, and the two settings are independent of each other. Perplexity uses PerplexityBot to crawl the pages it can cite, while Google uses Googlebot to index pages and the Google-Extended token to govern AI training. Because the crawling jobs are split this cleanly, one robots.txt file can allow the citation crawlers and disallow the training crawlers without conflict. For a fuller picture of how [AI search engines crawl sites differently](/signals/compare/ai-search-engine-crawling-differences) across ChatGPT, Perplexity, and Google, the per-engine differences matter more than any single blanket rule. ### The Costly Mistake Is Blocking Search to Stop Training The expensive mistake is disallowing a search crawler in an attempt to stop training, which quietly removes a brand from AI answers. The page still loads normally for people, so the lost citation stays invisible until someone checks. The cost is real because a large share of buyers now begin in AI answers: 51% of business-to-business (B2B) software buyers start their research with an AI chatbot rather than a traditional search engine ([G2](https://learn.g2.com/g2-2026-ai-search-insight-report)). In our work with B2B brands, an over-broad bot block is one of the most common reasons a page stops being cited. The fix is to target the training crawlers specifically, GPTBot and the Google-Extended token, rather than blocking all automated traffic. ## Which Crawlers to Allow for AI Citation AI citation requires keeping the search crawlers allowed, because those are the crawlers that read your pages and let AI answers surface and link them. The search crawlers that matter are OAI-SearchBot, PerplexityBot, and Googlebot. Allowing all three keeps you eligible to be cited across ChatGPT, Perplexity, and Google. ### Allow the Search Crawlers Allowing the search crawlers is what keeps each engine able to cite you. Allow OAI-SearchBot so your pages can appear and be cited in ChatGPT search answers. Allow PerplexityBot so Perplexity can surface and link your pages, and allow Googlebot so you stay indexed for Google Search and Google AI Overviews. These three are allow decisions you make deliberately, not defaults you can assume. A broad security rule or a catch-all Disallow can remove any of them without touching your training settings, which is why the allow list is worth stating explicitly. ### Remember Overviews Depend on Googlebot, Not Google-Extended Google AI Overviews are built from the Googlebot-crawled index, not from a separate AI crawler. The Google-Extended token controls whether your content trains or grounds Gemini, and it does not affect Search indexing or ranking. So disallowing Google-Extended keeps you eligible for Overviews while opting your content out of Gemini training. The distinction matters because Overviews reach an enormous audience: Google AI Overviews reach about 2 billion monthly users ([TechCrunch](https://techcrunch.com/2026/02/04/googles-gemini-app-has-surpassed-750m-monthly-active-users/)). Blocking Googlebot to avoid Gemini training would forfeit that reach, when the correct, narrower control is the Google-Extended token. ## Which Crawlers to Disallow for Model Training Model training is controlled by a different, smaller set of robots.txt directives than citation. To opt out, you disallow GPTBot and the Google-Extended token, the two controls the major engines expose for training. Because search runs on separate crawlers, these disallow rules do not remove you from AI answers. ### Disallow the Training Controls Disallowing the training controls is a narrow, deliberate action. Disallow GPTBot to opt your content out of training OpenAI's generative models, and disallow the Google-Extended token to opt out of Gemini training and grounding. Neither setting removes you from search, because ChatGPT search and Google Search read your pages through OAI-SearchBot and Googlebot instead. That separation is the whole point: the training disallows and the citation allows live in the same file and do not interfere. If you want the mechanics of how [llms.txt and robots.txt differ for AI optimization](/signals/compare/llms-txt-vs-robots-txt-ai-optimization), the controls described here all sit in robots.txt, which every major AI crawler reads. ### Decide User-Initiated Fetches Separately User-initiated fetches are a separate policy question from automatic crawling. Agents such as ChatGPT-User and Perplexity-User fetch a page because a person asked a question about it, and they may not follow robots.txt crawl rules the way automated crawlers do. Treat them as their own decision rather than folding them into your training or citation rules. Document which user-initiated agents you allow, so the overall access policy stays auditable and a later reviewer can see the reasoning. This matters most when a security team tightens bot rules and needs to know which agents were allowed on purpose. ## How to Set and Verify the robots.txt Rules robots.txt is where all of these decisions are set, one User-agent block at a time. The file lets you allow the search crawlers and disallow the training controls in the same place, then confirm the result in your logs. Implementation and verification are two steps, and skipping the second is how silent blocks survive. ### Set Per-Crawler Rules in One File Setting the rules means adding one User-agent block per crawler with the Allow or Disallow you intend. Keep your priority pages allowed for OAI-SearchBot, PerplexityBot, and Googlebot, and disallowed for GPTBot and the Google-Extended token. Avoid a broad catch-all Disallow under User-agent: *, which sweeps in the search crawlers along with everything else. Teams we work with often reach for a single blanket rule, which is exactly how a citation crawler gets blocked by accident. If you manage the file through a content management system (CMS), the steps for [configuring AI crawlers on WordPress and other CMS platforms](/signals/article/ai-crawler-configuration-wordpress) keep the per-crawler blocks intact. ### Confirm Citation Crawlers Still Reach You Confirming the citation crawlers still reach you is the step that proves the policy worked. After the change, check your server or content delivery network (CDN) logs for OAI-SearchBot, PerplexityBot, and Googlebot returning a 200 status on your priority pages, not a 403 or 429. Allow time for each engine to re-crawl and apply the new rules, since the change is not instantaneous. VisibilityStack's [Crawl Assurance Engine](/crawl-assurance-engine) checks whether AI citation and training crawlers reach your priority pages and flags the blocks that silently remove you from AI answers. Re-verify after any CDN or security change, so a firewall rule does not block a crawler you meant to allow. --- **Can I allow AI citations but block AI training?** Yes. The major engines use separate crawlers for each job, so you set the two decisions independently in robots.txt. Allow OAI-SearchBot, PerplexityBot, and Googlebot to stay citable in AI answers, and disallow GPTBot and the Google-Extended token to opt your content out of OpenAI and Gemini model training. **Which AI crawlers are for training and which are for search?** OpenAI's GPTBot is for training and OAI-SearchBot is for ChatGPT search. Perplexity's PerplexityBot is for search. Google's Googlebot indexes for Search and AI Overviews, while the Google-Extended token controls Gemini training. Allow the search crawlers and disallow the training controls to split the decision. **Does blocking GPTBot remove me from ChatGPT answers?** No, not by itself. GPTBot handles OpenAI model training, while ChatGPT search citations rely on OAI-SearchBot. Blocking GPTBot alone opts you out of training but can leave you citable through OAI-SearchBot. Blocking both removes you from ChatGPT's answers entirely, so keep OAI-SearchBot allowed. **Will disallowing Google-Extended hurt my Google visibility?** No. Google-Extended controls Gemini training and grounding and does not affect Search indexing or ranking. Google AI Overviews are built from the Googlebot-crawled index, so disallowing Google-Extended opts you out of Gemini training while keeping you eligible for Search and AI Overviews. Blocking Googlebot would hurt visibility. **Do user-initiated AI fetches follow robots.txt?** Often not. Agents like ChatGPT-User and Perplexity-User fetch a page because a person asked, so they may ignore robots.txt crawl rules. Treat them as a separate policy decision from automatic crawling, and document which user-initiated agents you allow so your overall access policy stays auditable. **How do I confirm my citation crawlers still work after the change?** Check your server or CDN logs for OAI-SearchBot, PerplexityBot, and Googlebot returning a 200 status on your priority pages after the robots.txt change. Allow time for each engine to re-crawl, and re-verify after any CDN or security change so a firewall rule does not block a crawler you meant to allow.