AI Crawler Configuration Guide for WordPress & CMS Platforms

Configure AI crawler access on WordPress by editing robots.txt to allow or block specific AI bots (citation bots like OAI-SearchBot, training bots like GPTBot), creating an llms.txt file to structure content, or using a WordPress plugin for easier management. Verify your security plugins aren't overriding rules, then track AI citations to measure impact.

AI crawler configuration is the process of allowing or blocking specific AI training and search bots on a WordPress site by editing robots.txt, creating llms.txt, using plugins, or setting DNS rules. The goal is to control whether AI engines can access your content for training, citation, or both.

In our work with B2B brands, crawler configuration is almost always the first technical fix that unblocks citation visibility. Most brands discover their security or SEO plugin has been blocking AI crawlers by default for months, and they never knew until they checked the access logs.

Why AI Crawler Configuration Matters for WordPress Sites

Citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) must be explicitly allowed in robots.txt to appear in AI-generated answers. Training bots (GPTBot, ClaudeBot, Google-Extended) crawl by default unless blocked; many brands choose to restrict training access while allowing citation.

If you don't configure AI crawlers on your WordPress site, you risk either blocking citation bots entirely (losing all AI visibility) or allowing training bots to ingest your content without consent.

Most WordPress sites ship with security plugins that block unfamiliar bots, and AI crawlers are still categorized as unfamiliar by WordFence, Sucuri, and many caching layers. That means your site may be invisible to ChatGPT, Perplexity, and Claude even though your content ranks well in Google.

The stakes are measurable. AI-search-referred visitors convert at roughly 4.4x the rate of traditional organic search visitors, and AI-referred traffic converts to sign-ups at about 1.66% versus 0.15% for organic search. If citation bots can't crawl your site, you don't appear in those answers, and you lose that conversion lift.

How to Understand AI Crawler Types and User Agents

AI crawlers split into two categories: citation bots and training bots. Citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) crawl your site to retrieve and cite content in real-time answers. Training bots (GPTBot, ClaudeBot, Google-Extended) crawl to ingest content for model training, not for live citation.

The user-agent strings you configure in robots.txt are:

Most B2B brands allow citation bots and block training bots. The rationale is simple: citation drives pipeline, training doesn't compensate you for ingesting your IP. Your configuration will depend on your brand's content licensing policy and your appetite for model training.

Step 1: Audit Your Current Crawler Access and Logs

Before you change anything, audit which AI bots are already crawling your WordPress site. Check three sources: Google Search Console's Crawl Stats report, your server access logs, and any security plugin logs.

Google Search Console: Navigate to Settings → Crawl Stats. Filter by host and look for user-agent strings like OAI-SearchBot, PerplexityBot, and Claude-SearchBot. If you see zero requests from these bots, they're either blocked or haven't discovered your site yet.

Server access logs: SSH into your server and grep your access logs for AI bot user agents. Most hosts store logs at /var/log/apache2/access.log or /var/log/nginx/access.log. Run:

grep -i "GPTBot\|OAI-SearchBot\|ClaudeBot\|Claude-SearchBot\|PerplexityBot\|Google-Extended" /var/log/nginx/access.log If you see 403 responses or no entries at all, something is blocking the bots. If you see 200 responses, the bots are crawling successfully.

Security plugin logs: Open WordFence, Sucuri, or your active security plugin and check the blocked-request log. Many plugins label AI crawlers as unknown or suspicious and block them by default. If you find blocked OAI-SearchBot or PerplexityBot requests, you'll need to whitelist those user agents in the plugin settings.

In our experience, most WordPress sites block citation bots unintentionally. The first audit almost always surfaces a security rule or firewall preset that treats these bots like scrapers. Once you know what's blocked, you can configure access deliberately rather than guessing.

Step 2: Edit or Create Your Robots.txt File

robots.txt must be located at the domain root (e.g., example.com/robots.txt), not inside /wp-content/ or subdirectories. If your WordPress site doesn't have a robots.txt file yet, create one. If you already have one, edit it to add AI crawler rules.

How to Create or Edit Robots.txt Manually

Connect to your WordPress site via SFTP or cPanel File Manager. Navigate to the root directory (where wp-config.php lives). If robots.txt exists, download it and open it in a text editor. If it doesn't exist, create a new file named robots.txt.

To allow citation bots and block training bots, add these lines:

User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /

If you want to allow all AI crawlers (citation and training), use:

User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: /

If you want to block all AI crawlers, use:

User-agent: OAI-SearchBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /

Save the file and upload it to your domain root. Then verify it's live by visiting yourdomain.com/robots.txt in a browser. You should see your new rules.

How to Edit Robots.txt Using a WordPress Plugin

If you're using Yoast SEO or Rank Math, navigate to the plugin's tools section and look for the robots.txt editor. Yoast: Tools → File Editor. Rank Math: General Settings → Edit robots.txt.

Add the same user-agent rules shown above. Save the changes. Then check yourdomain.com/robots.txt to confirm the rules are live. If they don't appear, your security plugin or caching layer may be serving a cached or overridden version.

How to Verify Your Robots.txt Rules

Use Google Search Console's Robots.txt Tester tool (Settings → Robots.txt Tester). Paste your robots.txt content into the tester. Enter a sample URL (e.g., /blog/your-post) and select the user agent (OAI-SearchBot, GPTBot, etc.). The tester will show whether that bot is allowed or blocked for that URL.

If the tester shows "Blocked" for a citation bot you meant to allow, check for conflicting rules earlier in the file. robots.txt processes rules top to bottom, and the first matching rule wins.

Step 3: Create an Llms.txt File to Structure Your Content for AI

llms.txt is optional and does not override robots.txt; it is a content manifest that improves AI crawl quality but does not grant or deny access. The file lives at the domain root (e.g., example.com/llms.txt) and lists the most important pages on your site, organized by topic or funnel stage.

Think of llms.txt as a sitemap for AI engines. It tells the engine which pages matter most and how they relate to each other. If your site has 500 pages, you can use llms.txt to surface the 50 that best represent your brand, products, and buyer questions.

How to Write an Llms.txt File

Create a plain text file named llms.txt. Add a brief site description at the top (1-2 sentences). Then list your most important URLs, grouped by category. Example:

# YourBrand - B2B SaaS platform for X ## Product Pages https://yourdomain.com/product/feature-a https://yourdomain.com/product/feature-b ## Use Cases https://yourdomain.com/use-cases/industry-x https://yourdomain.com/use-cases/industry-y ## Buyer Resources https://yourdomain.com/guides/how-to-choose-x https://yourdomain.com/pricing

Save the file and upload it to your domain root. Verify it's live by visiting yourdomain.com/llms.txt. AI engines that support llms.txt will use it to prioritize which pages to crawl and cite.

For a detailed comparison of how llms.txt and robots.txt work together, see Llms.txt vs Robots.txt: Best Practices for AI Search Optimization.

How to Handle WordPress Security Plugin Conflicts

WordPress security plugins (WordFence, Sucuri), SEO plugins (Yoast, Rank Math), and Cloudflare can override or conflict with your robots.txt rules. Here's how to fix the most common conflicts.

WordFence and Sucuri

WordFence blocks unfamiliar user agents by default. Navigate to WordFence → Firewall → Rate Limiting Rules. Add OAI-SearchBot, Claude-SearchBot, and PerplexityBot to the whitelist. Save the settings.

Sucuri has a similar whitelist in Security → Settings → Whitelisted IPs. You'll need to whitelist the IP ranges for each bot. OpenAI publishes its IP ranges at openai.com; Anthropic and Perplexity don't always publish theirs, so you may need to whitelist the user-agent string instead.

Yoast and Rank Math

Yoast and Rank Math both offer robots.txt editors, but they can conflict with manual edits if you've also edited the file via SFTP. If you edit robots.txt manually, disable the plugin's robots.txt editor to avoid overrides. In Yoast, go to SEO → Tools → File Editor and check whether your manual rules appear. If they don't, Yoast is overriding them.

Cloudflare

Cloudflare's default 'Block AI Crawlers' setting overrides robots.txt; disable it if you want AI engines to crawl your site for citations. Log into Cloudflare. Navigate to Security → Bots. Toggle "Block AI Scrapers and Crawlers" to off. This allows all AI bots; you'll still control which bots crawl via robots.txt.

If you leave the Cloudflare block enabled, your robots.txt allow rules won't matter. Cloudflare blocks at the DNS/firewall level, before the bot ever reaches your server.

How to Track AI Citations After Configuration

After configuration changes, AI citations typically appear or disappear within 2-4 weeks; use an AI-visibility platform to track the change. Manual citation tracking is impractical at scale because you need to fire your target prompts daily across multiple engines (ChatGPT, Perplexity, Claude, Google AI Overviews) and log every cited domain.

VisibilityStack tracks where your brand and domain are cited across AI engines and ties that to pipeline through the Inbound Conversion Score. It fires your mapped prompts daily, logs every citation, and alerts you when a page gains or loses citation coverage.

The Agentic Platform (Expert Guided) tier starts at $800/month; a Generative Engine Optimization (GEO) expert guides you at every step and runs the Demand Engineering System for you (the agents do the work, a dedicated strategist guides the calls and turns each report into a plan, your team stays at the controls). $800 is a deliberate floor, not a markup; the tier includes expert guidance, the Demand Engineering System doing the work, and a dedicated strategist guiding month over month.

Below it, the only honest offering is unguided automation, which doesn't move pipeline for a B2B brand. For a broader list of citation tracking options, see Best AI Brand Monitoring and Citation Tracking Tools.

How to Choose the Right Configuration for Your Brand

Your crawler configuration depends on three factors: your content licensing stance, your revenue model, and your competitive position.

If your revenue depends on buyer-stage content getting cited: Allow citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and block training bots (GPTBot, ClaudeBot, Google-Extended). This is the default for most B2B SaaS brands. You want AI engines to cite your guides, comparisons, and product pages, but you don't want your content ingested into foundational models without compensation.

If you publish open-source or community-driven content: Allow all AI crawlers (citation and training). Your content is already public, and model training can amplify distribution. Many developer tools and open-source projects take this stance.

If you sell proprietary research or gated content: Block all AI crawlers. You don't want your IP cited or trained on without a license agreement. This is common for analyst firms, legal publishers, and academic journals.

Most brands find the first option (allow citation, block training) delivers the best ROI. You gain AI visibility without licensing your IP to model providers. Once you configure access, audit your citations monthly and adjust your allow/disallow rules based on which engines drive qualified traffic.

For a step-by-step crawlability audit checklist, see Technical SEO Audit Playbook for AI Crawlers. If you need expert guidance on configuration and ongoing citation tracking, the Crawl Assurance Engine finds and prioritizes what blocks AI crawlers and citations: crawler access, indexability, canonical/duplicate pages, thin content, redirect chains, schema, and speed.