Who May Read the Site: A Practical AI Crawler Access Review for UK Firms

In 2026, the question for UK businesses is no longer whether AI crawlers exist. They appear in server logs, they influence whether a service page is cited in an AI-generated answer, and they can arrive in volume. What remains uncertain is whether they are being let in on purpose. Too often, access is decided by a plugin default, a firewall rule added after an attack, or a robots.txt file copied from another organisation and never reviewed.

An AI access audit is therefore a governance exercise before it is a technical one. It asks a simple commercial question: which agents should be able to read which parts of the site, and why?

Begin with the commercial question, not the user-agent list

Before inspecting a single line of configuration, UK firms should define what the website is for. A professional services firm may want its service pages, location pages and case studies read widely, because citation in an AI answer is a form of referral. A publisher may want retrieval but not training. A business with gated guides, client portals or proprietary research may want those areas closed to automated agents while keeping the public marketing pages open.

The allow-or-block policy flows from that decision. It should not be reverse-engineered from whatever a security plugin ships with. The audit team needs a short list of page types: public service pages, blog articles, location pages, resource libraries, PDFs, internal search results, faceted navigation, staging URLs and any area behind a login. Each needs a deliberate status.

Read robots.txt as instructions, not as a firewall

robots.txt remains the first place many firms look, but it is a crawl instruction rather than an access control mechanism. It can discourage compliant crawlers, yet it does not stop a determined agent from requesting a page. That distinction matters when a business assumes a robots.txt block is protecting sensitive material.

A practical review should check for:

• A wildcard Disallow: / rule that accidentally catches everything after a launch or migration.
• Separate user-agent groups for AI-related tokens such as GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended, CCBot and Bytespider.
• Rules that block CSS, JavaScript or image directories for search crawlers as well as AI agents.
• A sitemap directive that points to the current XML sitemap, not an old domain.
• Contradictions between robots.txt and HTTP headers such as X-Robots-Tag.

Google-Extended is a useful example of why the wording matters. It is a control token that lets publishers manage use of their content in certain Google AI products; blocking it does not remove pages from Google Search. UK firms should verify current documentation rather than rely on second-hand summaries, because the relationship between search crawling and AI use is not identical across providers.

Inspect the infrastructure layer: CDN, WAF and bot management

Many AI crawler blocks now live outside the CMS. A UK business may have a clean robots.txt and still be invisible to AI agents because a content delivery network, web application firewall or bot-management service is returning 403 or 429 responses.

Common culprits include:

• Bot Fight Mode or managed bot rules that classify unfamiliar agents as hostile.
• Rate-limiting rules designed for scraper defence but applied to all automated traffic.
• Browser integrity checks and JavaScript challenges that AI crawlers cannot complete.
• Geo-blocking rules that inadvertently exclude crawler IP ranges.
• IP allowlists that were configured for one search engine and never revisited.

This is especially relevant for UK ecommerce and professional services sites that tightened security after a period of automated attacks. A rule that stops credential stuffing can also stop a retrieval crawler. The audit should review firewall logs for blocked user agents and check whether the block is intentional. Where a CDN offers verified-bot categories, the firm should confirm which agents are recognised and whether custom rules override those categories.

Audit CMS defaults and plugin behaviour

Content management systems and their plugins add another layer of accidental control. Security plugins may block user agents by default. SEO plugins may add noindex headers to entire post types. Maintenance modes, coming-soon pages and password protection can remain active after a site goes live. Some hosts apply a staging-level restriction at the server level, which is invisible in the CMS admin.

A UK SME should check:

• Whether the site is publicly indexable and crawlable after the most recent launch or migration.
• Whether security or optimisation plugins have a bot-blocking feature enabled.
• Whether the XML sitemap includes the page types that should be discoverable.
• Whether the CDN is caching a block page and serving it to allowed agents.
• Whether .htaccess or nginx configuration contains user-agent rules that are not visible in the CMS.

Test as the crawler would, then read the logs

Configuration review should be paired with direct testing. A simple request using a known AI user-agent string can reveal whether the server returns a 200 response, a redirect, a challenge page or a block. Server logs then show which agents are actually arriving and which are being turned away. For Googlebot, Search Console URL Inspection remains useful; for AI agents, log analysis is usually the more honest source.

The test should also confirm that important pages return the correct status code and that robots.txt itself returns a 200 response. A missing robots.txt can be interpreted differently by different agents, and a redirect on robots.txt can create confusion. Response headers should be checked for X-Robots-Tag: noindex and for cache directives that might serve an old block to a new crawler.

Separate search-critical resources from AI-only decisions

One of the most common technical errors is blocking CSS and JavaScript in robots.txt while expecting Google to render the page correctly. Search-critical resources should remain available to search crawlers. AI crawlers often do not execute JavaScript, so core text should be present in the initial HTML response; relying on client-side rendering for essential content is a visibility risk regardless of AI policy.

A sensible approach is to keep search-critical assets open and make page-level decisions for AI access. Citation-friendly pages such as service descriptions, location pages and original explainers can be allowed. Thin archives, internal search results, faceted navigation and duplicate PDFs can be blocked or consolidated. If the business licenses content, the policy should align with its terms rather than relying on robots.txt alone.

Write a one-page access policy and keep a register

The output of the audit should be a short, dated document that names an owner. It should record the user-agent groups reviewed, the allow-or-block decision for each page type, the infrastructure controls in place, and the review cadence. It should be shared with sales, marketing and legal where relevant, because AI access affects lead generation and content licensing as much as it affects technical SEO.

A quarterly review is reasonable for most UK firms, with an additional check after any CMS migration, CDN change or security incident. The register should note when a plugin update or firewall change could alter the guest list. Without that record, the business is relying on memory and vendor defaults.

The guest list is a business decision

AI crawlers are not a passing technical curiosity. They are part of the audience now. A UK website that cannot be read by the agents a business wants to reach is invisible in a growing channel; a site that lets every agent read everything may be giving away value it never intended to share. The audit is not about hostility to AI. It is about making the door policy explicit, testing it, and reviewing it as the commercial strategy changes.