Make sure robots.txt does not block the answer crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews and AI Mode) and Bingbot (Copilot). Training crawlers such as GPTBot, ClaudeBot and CCBot are your choice: blocking them costs no answer visibility. Google-Extended is a trade, because it also controls Gemini answers. Every named group needs its own private-area rules. Then check that your CDN or firewall lets the answer crawlers through.
The complete robots.txt: copy this directly
Save this file as robots.txt in the root directory of your website (for example yourdomain.ie/robots.txt). Replace the sitemap URL on the final line with your own. Note that every group repeats the private-area rules: crawlers read robots.txt group by group, and a bot with its own named group ignores the rules under User-agent: * (RFC 9309).
Why each bot matters
Each AI platform uses its own named crawlers, and most now separate answering from training. Crawlers treat your pages as open unless a rule blocks them, so the Allow lines above are about clarity. The reverse is what matters: if an answer crawler is blocked, that platform cannot cite your page, however good it is. The roles below are from each provider's own documentation (OpenAI, Anthropic, Perplexity, Google).
| Bot name | Platform | What it powers | Status |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Answer: builds the index ChatGPT search has used since September 2026. Blocked sites are not shown in ChatGPT search answers. | Allow |
| ChatGPT-User | OpenAI | Answer: fetches a page live while ChatGPT answers. Cloudflare classes it as an Agent, which it blocks by default on some new sites. | Allow |
| Claude-SearchBot | Anthropic | Answer: indexes pages for Claude's search answers | Allow |
| Claude-User | Anthropic | Answer: fetches a page when a Claude user's question needs it | Allow |
| PerplexityBot | Perplexity AI | Answer: builds Perplexity's own index; not used to train foundation models | Allow |
| Googlebot | Answer: Google Search, AI Overviews and AI Mode all read Google's normal index | Allow | |
| Bingbot | Microsoft | Answer: Bing search, Copilot and Bing's AI summaries; ChatGPT uses Bing mainly in Deep Research | Allow |
| Google-Extended | Hybrid: Gemini training and Gemini grounding together. Blocking it removes you from Gemini answers, never from Search, AI Overviews or AI Mode. | Allow, unless you accept losing Gemini | |
| GPTBot, ClaudeBot, CCBot | OpenAI, Anthropic, Common Crawl | Training only: collect pages for future models, not for answers | Your choice: blocking costs no answer visibility |
Two older Anthropic names, anthropic-ai and Claude-Web, still appear in many robots.txt templates. Anthropic has retired them in favour of ClaudeBot, Claude-SearchBot and Claude-User. Other answer crawlers worth allowing are Applebot (Siri and Spotlight), Meta-WebIndexer (Meta AI) and Amzn-SearchBot (Alexa). xAI publishes no crawler documentation, so no robots.txt line can promise coverage in Grok.
Can I allow AI search bots but block AI training bots?
Yes. The answer crawlers and the training crawlers are now separate, so this is a policy decision, not a visibility one. OpenAI separates its crawlers by purpose:
- GPTBot: used only to train OpenAI's models. You can block it if you object to training use, and you lose no ChatGPT answers.
- OAI-SearchBot: builds ChatGPT's search index. Blocking it removes you from ChatGPT search answers, and OpenAI says a change takes about 24 hours to take effect.
Anthropic mirrors this split: ClaudeBot is the training crawler, while Claude-SearchBot powers Claude's search citations.
Google-Extended works differently, and it is not a free training opt-out. It is a token Google reads from robots.txt to decide whether your content can train Gemini and ground answers in the Gemini app. Blocking it removes you from Gemini answers. It does not remove you from Google Search, AI Overviews or AI Mode, which follow your normal Googlebot rules (Google).
Whatever you decide on training, never block an answer crawler by accident. For most businesses the simplest file allows everything except private areas. If you do block training, block the training-only names and leave Google-Extended alone unless you are content to disappear from Gemini.
How to verify your robots.txt is working
- Visit the file in your browser. Go to
yourdomain.ie/robots.txt. If you see the file contents, it is in the right location. A blank page or 404 means the file is missing or misplaced. - Check Search Console's robots.txt report. In Search Console go to Settings → robots.txt to confirm Google can fetch and parse your file. Google retired its old per-bot tester in 2023, so to test a specific user agent (for example OAI-SearchBot) use the robots.txt tester in Bing Webmaster Tools or a reputable online checker. Test each named group: a bot with its own group ignores the wildcard rules.
- Check your server or CDN logs, and the status codes. Look for OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. A request is only useful if it got a 200: a 403 or a challenge page means your CDN or firewall blocked it, whatever robots.txt says. Repeat this every month.
- Set up Bing Webmaster Tools and Search Console. Bing's AI Performance report (public preview since February 2026) shows a sample of your citations in Copilot and Bing's AI summaries, with no clicks. Search Console's Generative AI report (since 31 August 2026) shows your impressions in AI Overviews and AI Mode. Neither covers ChatGPT.
Common mistakes that block AI crawlers without you realising
- A blanket
Disallow: /underUser-agent: *. Some WordPress security plugins or hosting defaults add this, which blocks every crawler on the web, including all AI bots. Check your current robots.txt before making changes. - Noindex or nosnippet applied too broadly. A
<meta name="robots" content="noindex">tag on a key page removes it from the index.nosnippet,data-nosnippetand a lowmax-snippetdo the same job more quietly: Google only uses indexed, snippet-eligible pages in AI Overviews and AI Mode (Google). - A named group that forgets the wildcard rules. A bot with its own
User-agentgroup follows only that group. Add a group for GPTBot with justAllow: /, and GPTBot may now crawl your private areas. Repeat the private disallows in every group. - Trusting robots.txt alone. robots.txt is a request; a CDN or firewall enforces its own blocks. Your CDN may also serve a cached copy of the old file. Clear the cache after changes, and check the CDN's bot settings as well as the file.
- Wrong file location. robots.txt must live at the root of your domain (
yourdomain.ie/robots.txt), not in a subfolder. If your CMS installs it elsewhere, it will be ignored.
Before making any changes, visit yourdomain.ie/robots.txt and check two things: whether any rule blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot, and whether a named group lacks your private-area rules. Then open your CDN's bot settings. An accidental block, in the file or at the CDN, is one of the most common and most easily fixed causes of AI search invisibility.
What else affects whether AI systems can access your content
robots.txt asks crawlers to stay out; it does not enforce anything. Since 15 September 2026, Cloudflare blocks its Training and Agent crawler categories by default on ad-bearing pages for new sites, and it classes ChatGPT-User as an Agent (Cloudflare). So a perfect robots.txt can still return a 403. AI assistants also cannot use your content if:
- Your key pages require a login or are behind a paywall
- Content is rendered entirely by JavaScript with no server-side HTML (GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files but do not run them, per Vercel and MERJ, December 2024)
- Your hosting provider blocks bots at the server or firewall level before they reach robots.txt
- Cloudflare or another firewall challenges or blocks AI crawlers, by default or by a rule your security team turned on
- Your site requires specific cookies or sessions to display content
One file you do not need is llms.txt. SE Ranking's study of 300,000 domains (October 2026) found no link to AI citations, and Google says no AI text files are needed. An AI Visibility Audit checks all of these factors, not just robots.txt, to show what AI assistants can and cannot reach on your site.
Frequently Asked Questions
Does blocking GPTBot stop my site appearing in ChatGPT?
No. GPTBot only controls whether OpenAI can use your pages to train future models. OAI-SearchBot builds the index that ChatGPT search has used since September 2026: block it and OpenAI says your pages will not be shown in ChatGPT search answers. ChatGPT-User fetches pages live while ChatGPT answers, so check that your CDN does not block it either.
What happens if I have no robots.txt file at all?
Most AI crawlers default to treating all pages as accessible when no robots.txt file exists, the same as having a fully permissive file. However, an explicit, correctly configured robots.txt is best practice: it confirms your crawl permissions clearly, includes your sitemap URL, and lets you specify which private areas to protect.
Can I allow AI search bots but block AI training bots?
Yes. Block GPTBot, ClaudeBot and CCBot and allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Refusing training then costs no answer visibility. Google-Extended is the exception: it controls Gemini grounding as well as training, so blocking it removes you from Gemini answers, though never from Google Search, AI Overviews or AI Mode.
How do I verify AI crawlers can access my site?
Check your server or CDN logs for requests from OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, and look at the status codes: you want 200, not 403 or a challenge page. Confirm Google can read your file in Search Console's robots.txt report, and use the robots.txt tester in Bing Webmaster Tools for a specific user agent. Repeat the log check every month, because CDN and firewall rules change without notice.
Does my WordPress site need a separate robots.txt plugin?
WordPress generates a virtual robots.txt automatically if no physical file exists, but it may not include AI crawlers. The safest approach is to create a physical robots.txt file in your site's root directory. A physical file always takes precedence over WordPress's virtual one. Plugins like Yoast SEO allow you to edit robots.txt from inside WordPress if you prefer.