We checked 93 UK private surgeon websites against nine AI search basics: the things that affect whether AI assistants can find, read and cite a site. Only 1 passed every check that applied to it, and 52 passed half or fewer. Almost every site let AI crawlers in. The gaps were in what those crawlers found: 45% had no structured data describing the practice.
The results at a glance
| Check | What it means | Sites passing |
|---|---|---|
| AI search crawlers allowed | ChatGPT, Perplexity and Claude search crawlers can read the site | 87 of 93 (94%) |
| No Cloudflare AI block | No Cloudflare-added rule blocking AI crawlers | 93 of 93 (100%) |
| llms.txt present | A plain-text site summary for AI tools | 21 of 93 (23%) |
| Practice schema | Organization, Physician or MedicalClinic markup on the home page | 51 of 93 (55%) |
| FAQs with matching schema | Visible FAQs with matching FAQPage markup | 12 of 93 (13%) |
| Named authors and dates | Articles show who wrote them and when | 14 of 39 sites with articles (36%) |
| Home page title and description | Both written for the page | 75 of 93 (81%) |
| Condition pages answer first | The first paragraph answers the page’s question | 63 of 79 sites with condition pages (80%) |
| Mobile speed | Lighthouse mobile performance score of 90 or above | 12 of 93 (13%) |
How we audited 93 surgeon websites
The sample. We ran about 100 web searches of the form [specialty] surgeon [city] private in October 2026, across 14 specialties and subspecialties (from knee and spine to ENT, gynaecology, vascular and breast surgery) and 19 UK cities. We kept websites belonging to an individual consultant or a practice of up to six consultants, and excluded hospital groups, directories, NHS pages, clinic chains and sites run by our own clients. That gave 99 sites. On review we removed 6 more: 4 were profile pages built by a directory rather than the surgeon’s own site, 1 was a larger group, and 1 domain now hosts an unrelated gambling blog. That left 93. Where one practice appeared for several searches, we counted it once.
The dates. All checks were run on 8 October 2026. Websites change, so every result describes the site on that day.
The tools. A script read each site’s live robots.txt, /llms.txt and home page HTML, and parsed the structured data on the home page and on each site’s FAQ and condition pages. To catch blocking that robots.txt does not show, we also requested each home page with each AI crawler’s published user agent and compared the response with a normal browser request. Speed was measured with Google’s Lighthouse (version 13.5) on its simulated mobile setting, three runs per site, taking the median. Condition pages, articles and FAQs were reviewed page by page against a written scoring rule, and borderline sites were re-scored by a single reviewer so the rule was applied the same way throughout.
Anonymity. No site is named or identifiable in this post, and results are reported in aggregate only.
AI search crawlers allowed in robots.txt
What we checked: whether each site’s robots.txt allows the crawlers that fetch pages for AI search answers: OpenAI’s OAI-SearchBot, PerplexityBot, and Anthropic’s Claude-SearchBot. We recorded separately whether sites block the training crawlers GPTBot, ClaudeBot and Google-Extended.
Why it matters for AI search: OpenAI’s documentation says OAI-SearchBot is used to surface websites in ChatGPT’s search features, while GPTBot collects content for model training. Perplexity says PerplexityBot surfaces and links websites in its results and is not used for training. Anthropic separates ClaudeBot (training) from Claude-SearchBot (search). Google states that blocking Google-Extended does not affect inclusion in Google Search. Blocking a training crawler is a legitimate choice. Blocking a search crawler keeps the site out of that assistant’s answers. Across about 140 million websites, Ahrefs found 5.89% block GPTBot.
What we found: no site blocked any AI crawler in its robots.txt, search or training. 4 sites had no robots.txt at all, which also means no restriction. The server-level test told a slightly different story. 6 of 93 sites (6%) refused requests identifying as at least one AI search crawler, and 12 (13%) refused GPTBot. The same sites’ robots.txt files allow these crawlers, so the blocking most likely comes from a security plugin or hosting firewall rather than a deliberate choice. Our requests did not come from the crawlers’ own networks, so treat these figures as indicative.
How to fix it: if any search crawler is disallowed in your robots.txt, ask your developer to allow it. Also ask whether a security plugin or your host blocks bots by user agent. Whether to block training crawlers is your decision.
Cloudflare AI bot blocking
What we checked: whether sites served through Cloudflare carry its managed AI-crawler block in their robots.txt. Firewall-level blocking is not visible from outside, so this check undercounts.
Why it matters for AI search: Cloudflare’s managed robots.txt disallows a list of AI crawlers including GPTBot, ClaudeBot and Google-Extended, and is prepended to the site’s own file. Since July 2025, Cloudflare has blocked AI crawlers by default on new domains.
What we found: 23 of 93 sites (25%) were served through Cloudflare. None carried the managed AI-crawler block. Two carried Cloudflare’s content-signal licensing text, which sets out terms but blocks nothing.
How to fix it: ask whoever manages your domain whether it runs through Cloudflare, and if so, to check the AI crawler settings.
llms.txt present
What we checked: whether /llms.txt returns a plain-text or markdown file.
Why it matters for AI search: llms.txt is a proposal from Jeremy Howard, first published in September 2024, for a short file describing a site to AI tools. It is not a standard. Google says no AI text files or special markup are needed to appear in AI Overviews or AI Mode. We include it because it costs little to add.
What we found: 21 of 93 sites (23%) had an llms.txt file. 18 of those 21 had been generated automatically by the site’s platform or SEO plugin, such as Wix, Yoast, Rank Math or All in One SEO, rather than written for the practice. If your site has one, read it to check it describes your practice accurately.
How to fix it: a short file describing your practice, with links to your key pages. Lower priority than the crawler checks.
Organization, Physician or MedicalClinic schema
What we checked: whether the home page carries valid structured data describing the practice, using the schema.org types Organization, Physician or MedicalClinic.
Why it matters for AI search: Google describes structured data as explicit clues about the meaning of a page. It states your name, specialty and locations in a form search systems do not have to infer. Google also says no special schema is needed for AI features, so this check is about accuracy, not a shortcut.
What we found: 51 of 93 sites (55%) had structured data describing the practice. 20 had some structured data, usually generated by a plugin to describe the web page, but nothing about the practice itself. 22 had none at all. 2 had markup that could not be read because of errors.
How to fix it: add Physician or MedicalClinic markup with your name, specialty, practice addresses and GMC-registered name, then test the home page in the Rich Results Test.
FAQ content with FAQPage schema
What we checked: whether the site has visible FAQ content, and whether it carries FAQPage markup that matches the visible questions and answers.
Why it matters for AI search: a question followed by a short, self-contained answer is easy for an AI system to lift and quote. Google now shows FAQ rich results only for well-known, authoritative government and health websites, so most surgeon sites will not see the visual result. Google’s AI features guidance asks that any structured data match the visible text.
What we found: 41 of 93 sites (44%) had visible FAQ content. 12 of those had FAQPage markup matching the questions on the page.
How to fix it: add an FAQ block to each condition page using questions patients ask you in clinic, then mark it up.
Named author and visible updated dates on articles
What we checked: for sites with a blog or news section, whether a sample article shows a named author and a visible published or updated date.
Why it matters for AI search: Google’s helpful content guidance encourages accurate bylines and warns against changing dates when content has not substantially changed. Ahrefs’ research, summarised in their AEO video, found AI-cited content is on average 25.7% fresher than content ranking in Google.
What we found: 39 of the 93 sites had articles. On the most recent article, 17 showed a named author, 22 showed a date, and 14 showed both. Bylines reading admin or a username did not count. On two sites, the most recent posts were spam published under the surgeon’s name, and on another the only post was an unedited template with placeholder text.
How to fix it: put your name and the date the article was last reviewed at the top of every article, and update the date only when you change the content.
Unique title tag and meta description on the home page
What we checked: whether the home page has a title tag and meta description written for it, rather than missing, blank or left as a template default.
Why it matters for AI search: these are the first lines a search system reads about a site. Google advises unique descriptions for each page.
What we found: 75 of 93 sites (81%) passed. Every site had a usable title. The 18 that failed had no meta description on the home page.
How to fix it: a title with your name, specialty and location, and a one-sentence description of who you treat and where.
Condition pages that answer a question in the first paragraph
What we checked: on up to three condition or procedure pages per site, whether the first paragraph says what the condition or procedure is and who it is for. Pages opening with a welcome, a biography or a statement about the practice failed.
Why it matters for AI search: AI systems often quote a short passage rather than a whole page. Ahrefs’ AEO guidance recommends putting the answer first, in sections that make sense on their own.
What we found: 79 of 93 sites had at least one condition or procedure page. Of those, 63 (80%) passed, with at least two of the three pages reviewed opening with an answer. The pages that failed most often opened with the surgeon’s credentials, a description of the practice, a disclaimer or a success statistic. 14 sites had no condition pages at all: their conditions appeared only as a list of names, in downloadable PDFs or on a single-page site.
How to fix it: rewrite the first paragraph of each condition page as a 40 to 60 word answer, then add the detail below it. Our SEO and content service writes condition pages this way.
Mobile page speed
What we checked: the Lighthouse performance score for each home page on mobile, median of three runs.
Why it matters for AI search: Google lists page experience among the search basics that still apply to its AI features. Lighthouse, which produces the score, rates 0 to 49 as poor, 50 to 89 as needs improvement and 90 to 100 as good, and notes that scores vary between runs.
What we found: the median score was 69. 12 of 93 sites (13%) scored 90 or above, 80 scored between 50 and 89, and 1 scored below 50. Lighthouse run locally can score differently from Google’s PageSpeed Insights, so compare your own site using the same tool each time.
How to fix it: run your home page through PageSpeed Insights and send the report to your developer.
Limitations
93 sites is a sample, not a census, and one built from web searches leans towards practices that already appear in search results. Every result is a snapshot from 8 October 2026. Passing these checks makes a site readable to AI systems. It does not make an assistant cite or recommend it. In a study of 75,000 brands, Ahrefs found mentions across the web had a stronger link with appearing in Google’s AI Overviews than backlinks did. This audit does not measure mentions.
In my experience: Sam Stratton
Which of these gaps do you find most often when you first open a surgeon’s site, and which one takes longest to fix?
Most of the technical gaps in this audit are quick to fix. Allowing a crawler, writing a meta description or adding practice schema is usually a small job for whoever manages the site. What I’d check first is anything a plugin has set up on your behalf, such as an llms.txt file, a security rule that blocks bots, or schema describing the wrong kind of business, because nobody tends to look at those once they’re switched on. The slow part is content. Condition pages that answer the patient’s question in the first paragraph need the surgeon’s input and sign-off, and that takes far longer than the technical work.
Frequently asked questions
How do I check whether AI crawlers can read my website?
Open yoursite.co.uk/robots.txt in a browser. If OAI-SearchBot, PerplexityBot or Claude-SearchBot is disallowed, or User-agent: * is followed by Disallow: /, at least one AI search tool is told not to read your site.
Should I block GPTBot?
That is your choice. GPTBot collects content for training OpenAI’s models. OpenAI uses a separate crawler, OAI-SearchBot, for ChatGPT’s search results, so blocking GPTBot alone does not remove you from ChatGPT search.
Do I need llms.txt to appear in AI answers?
No. It is a proposal, not a standard, and Google says no AI text files are needed for its AI features.
Does schema markup get my practice into AI Overviews?
Not on its own. Google says no special schema is needed. Schema helps search systems read your name, specialty and locations accurately.
If you are deciding whether you need your own site at all, see our comparison of your own website and a hospital profile.
Want these checks run on your site? We’ll run the same checks on your website and send you the results, with a short note on what to fix first. It’s an assessment, not a sales call. Request a free audit.