Yes, your website can be turning away AI assistants even if your pages look fine in normal search. The two main places to check are your AI welcome sign, called robots.txt, and your firewall or CDN rules, which can block AI visitors before they ever see that sign. Since Cloudflare began blocking some training and agent categories by default on ad-bearing pages for new zones on 15 September 2026, a site can allow search engines in robots.txt and still return a 403 blocked response to ChatGPT-User, according to Cloudflare’s public documentation and product behaviour noted by HelloWorld on 24 October 2026. If you want citations and recommendations from AI assistants, you need both layers to allow the right AI visitors.
AI visibility now depends on access, not just content
If an AI visitor can’t reach your pages, your content can’t be cited. That sounds obvious, but it’s now a common problem.
Many businesses check only their AI welcome sign. They forget the firewall, the CDN, or a security plugin. Those tools can block AI visitors first. In practice, that means your site may be open to Google Search and Bing Search, but closed to some AI assistants.
This matters more now because the access paths are no longer the same. OpenAI search has run on its own index, called Labrador, since 4 September 2026, with scraping of search results used as a fallback and Bing used mainly in Deep Research, according to HelloWorld’s reviewed methodology dated 24 October 2026. So being in Bing’s phone book is no longer proof that ChatGPT can reach or cite you.
For business owners, the takeaway is simple. If you want to appear in AI answers, product picks, comparisons, or local recommendations, you must check both your welcome sign and your gatekeeper.
Your AI welcome sign and your firewall do different jobs
Your AI welcome sign, called robots.txt, is the small file that tells AI readers whether they’re invited in. Your firewall or CDN is the security guard at the door. It decides who gets through.
The AI welcome sign controls permission
This file sits at the root of your site, usually at /robots.txt. It tells search engines and some AI visitors which areas they may read and which areas they should avoid.
If you accidentally disallow key sections, you can hide your pricing, product, service, help, or comparison pages from AI readers. That can damage your chances of being cited.
It’s worth remembering that this file is only one layer. It’s a request, not a guarantee of access.
The firewall controls entry
Your firewall, CDN, host security layer, or plugin can block a visitor completely. When that happens, the AI visitor never gets to read your welcome sign.
A common blocked response is 403 Forbidden. In plain English, that means the visitor reached the door and was turned away.
This is why technical access now deserves monthly checking. HelloWorld treats monthly crawler-access monitoring as a standard step, following the changes reviewed on 24 October 2026.
Cloudflare changed the risk profile in September 2026
Cloudflare made firewall checks much more important for AI visibility.
Since 15 September 2026, Cloudflare has blocked its Training and Agent crawler categories by default on ad-bearing pages for new zones, while Search remains allowed, according to HelloWorld’s reviewed methodology dated 24 October 2026. ChatGPT-User is classed as an Agent. If a crawler has several roles, the most restrictive rule applies.
The practical effect is important. You can have a perfect AI welcome sign and still block ChatGPT-User at the CDN level. That means your content may remain visible in traditional search while becoming harder for some AI assistants to fetch directly.
HelloWorld’s view is clear here. The CDN step now carries the same weight as the robots.txt step. Existing zones may or may not have been switched, and that remains unconfirmed, so every business should check rather than assume.
You should allow answer and search visitors, not blindly allow training visitors
Training crawlers and answer crawlers are different. You should not treat them as the same thing.
If your goal is to be cited and recommended by AI assistants, the key issue is whether answer and search visitors can access your important pages. Blocking all AI visitors can cut you off from visibility. Allowing every possible training visitor without review may not match your policy, legal, or commercial position.
That is why broad advice like “block AI bots” or “allow all AI bots” is poor advice. You need a controlled access policy. It should reflect your business goals.
If you’re unsure where to start, focus first on the pages that drive revenue or trust. That includes service pages, product pages, pricing, comparison content, reviews, help content, returns information, location pages, and named author pages.
For related reading, see What is an llms.txt file and does your business need one. Also see How to measure your AI search citation performance.
The fastest check is to test four things
You don’t need a full technical audit to spot the main problems. You do need a structured check.
Check 1: Read your AI welcome sign
Open yoursite.com/robots.txt in a browser. Look for any lines that disallow important sections. Pay close attention to /products/, /services/, /pricing/, /blog/, /guides/, /help/, /locations/, and PDF folders.
If you see broad rules like “Disallow: /” for a named AI visitor, that is a hard stop. If you see rules that block CSS, JavaScript, or image folders, that can also cause problems because some systems need those files to render pages properly.
Check 2: Test whether the page returns 200 or 403
A healthy public page should usually return a 200 OK response. A blocked page may return 403 Forbidden.
Your developer can test this with command-line tools, server logs, or firewall logs. If you use Cloudflare, check Security, Bots, WAF rules, and any custom rules. If you use Wordfence, Sucuri, Imperva, AWS WAF, or host-level security, check those too.
You are looking for rules that challenge or block AI visitors by user agent, category, geography, rate limit, or threat score.
Check 3: Compare search access with AI assistant access
A page being in Google’s phone book or Bing’s phone book does not prove that all AI assistants can fetch it now.
Google’s own reporting changed on 31 August 2026, when Search Console’s Generative AI performance report went live worldwide, showing impressions inside Google’s AI answer box and AI Mode by page, country, date, and device, according to Google Search Console and HelloWorld’s reviewed methodology dated 24 October 2026. It shows no clicks and does not separate the two features.
Bing also offers AI reporting. Bing Webmaster Tools released its AI Performance report in public preview on 1 February 2026, with query-to-page mapping added on 23 March 2026, according to Microsoft Bing Webmaster Tools and HelloWorld’s reviewed methodology dated 24 October 2026. It provides total citations, cited pages, and grounding queries as a sample.
If your pages are indexed but have weak or missing AI impressions and citations, access is one of the first things to test.
Check 4: Review the pages that matter commercially
Do not stop at the home page. AI assistants often cite deep pages.
Check product pages, service pages, pricing pages, alternatives pages, review pages, support content, local landing pages, and key PDFs. HelloWorld now treats pricing and alternatives pages as high-weight levers because they are useful to buyers and easy for AI assistants to cite when accessible.
If you sell products, also review your feeds and merchant setup. See How ChatGPT shopping picks products, and why product feeds now matter and Google Merchant Center for AI shopping: a checklist for Irish retailers.
The most common mistakes are simple and expensive
Most access problems are not advanced. They’re overlooked defaults, old rules, or over-defensive security settings.
Blocking whole sections by accident
This often happens after a site migration, redesign, or staging setup. A developer may carry over a temporary block and forget to remove it.
Blocking by user agent without real review
Some plugins and firewall templates use broad bot-blocking lists. That can catch legitimate AI visitors along with bad traffic.
Allowing search, but blocking agents
This is now a specific risk on Cloudflare for some setups. Search may remain allowed while agents are blocked. That split matters because AI assistants don’t all fetch content the same way.
Forgetting PDFs and resource files
Many manufacturers, merchants, and building product firms keep the facts in PDFs. If those files are blocked, the assistant misses the detail. This is one reason technical product information often fails to appear in AI answers. See Why construction product PDFs are invisible to AI assistants.
Assuming schema or IndexNow will fix access
Your machine-readable business card, called schema, can help machines understand your page, but it does not override a block. IndexNow can help Bing and Copilot discover updates, but it is not a ChatGPT lever since OpenAI search moved to its own index on 4 September 2026, according to HelloWorld’s reviewed methodology dated 24 October 2026. For background, see What is schema markup, and does it matter for AI search? and What is IndexNow, and which AI assistants does it help?.
A practical checklist will catch most access issues
Use this list with your developer, agency, or hosting provider. It’s designed for non-technical business owners who need a clear pass or fail answer.
- Open your AI welcome sign at /robots.txt and save a copy for review.
- Check that key commercial sections are not disallowed, including product, service, pricing, review, comparison, support, location, and guide pages.
- Check that important PDFs are not blocked.
- Confirm that public pages return 200 OK, not 403 Forbidden, for normal visitors.
- Review Cloudflare bot settings, WAF rules, custom rules, and security events if you use Cloudflare.
- Review host security, plugin firewalls, and rate-limiting rules if you do not use Cloudflare.
- Ask your developer to test access for named AI visitors that matter to your business goals.
- Separate answer and search access from training access in your policy. Do not block both by accident.
- Re-check access after any redesign, migration, CDN change, or security update.
- Monitor AI visibility monthly using Search Console’s Generative AI report and Bing Webmaster Tools AI Performance report.
- Use Bing Webmaster Tools Topics Portfolio beta, added on 13 October 2026, to find grounding queries where competitors are cited and you are not, according to Microsoft Bing Webmaster Tools and HelloWorld’s reviewed methodology dated 24 October 2026.
- Document who approved the current access policy, and why, so it is not changed casually later.
The right policy is commercial, not ideological
You don’t need to take an all-or-nothing position on AI access. You need a policy that supports your business.
If your company wants more AI citations, more product inclusion, or more local recommendations, then answer and search access should be treated as a revenue issue. If your legal team has concerns about training access, those can be handled separately.
This distinction matters for boards, founders, and heads of marketing. It turns a vague AI debate into a practical decision. Which AI visitors help us win business, and are we allowing them through?
For local firms, the same principle applies. A restaurant, hotel, solicitor, or accountant can lose recommendation visibility if key pages are blocked. If that is relevant to you, see How Galway hotels can appear in AI travel recommendations or How Waterford accountants can be found in AI search.
Measurement should confirm access, not replace it
Reports can tell you what happened. They do not fix the cause.
GA4 added a default AI Assistant channel on 13 May 2026 for referrals from ChatGPT, Gemini, DeepSeek, Copilot and Grok, according to Google Analytics documentation and HelloWorld’s reviewed methodology dated 24 October 2026. It excludes Google AI features, is not retroactive, and misses some assistants such as Perplexity and Claude from the published list. That is why HelloWorld treats the number as a floor, never the KPI.
The stronger KPI is visibility by page in first-party reports, then citation coverage against the queries that matter. Access checks support that work. They do not replace answer content, reviews, comparison pages, local profiles, or feeds. They simply make the rest possible.
Fix access first, then improve the pages AI assistants cite
Once access is open, the next step is to make your pages worth citing.
That means visible question-and-answer text, clear pricing where appropriate, honest comparison pages, named authors, real dates, sourced figures, and strong review signals. It does not mean rewriting copy “for AI”. Google’s own generative AI optimisation guidance says that is unnecessary, and that content chunking is not required because its systems understand multi-topic pages, according to Google Search Central guidance cited in HelloWorld’s reviewed methodology dated 24 October 2026.
If you want the short version, do this in order: allow the right visitors in, make your key pages easy to understand, and measure where citations appear or fail.
Can robots.txt alone block AI assistants?
Yes, it can block some AI readers if your AI welcome sign tells them not to enter. But it is not the only risk. Your firewall or CDN can block them before they even read that file, which is now a major issue on some Cloudflare setups since 15 September 2026.
What does a 403 error mean for AI visibility?
A 403 Forbidden response means the visitor reached your site and was refused entry. If an AI visitor gets a 403 on a key page, that page is unlikely to be fetched directly for answers or citations.
Should I block all AI bots to protect my content?
That is usually too blunt. Training visitors and answer visitors are different. If you block both, you may also block the AI assistants that could cite, recommend, or send traffic to your business.
Does being indexed in Bing or Google prove AI assistants can access my site?
No. Being in Google’s phone book or Bing’s phone book shows search indexing, not full AI assistant access. This matters even more since OpenAI search began using its own index from 4 September 2026, according to HelloWorld’s reviewed methodology dated 24 October
