Enterprise Website Operations, in the Age of AI
Enterprise website operations is the discipline of keeping one site legible to three consumers at once: a human in a browser, a search crawler, and a retrieval system that fetches over HTTP, executes no JavaScript, and lifts passages rather than reading pages. On 9 September 2026 we fetched the seven domains Google ranks on page one for a core enterprise term, and 13 of 35 requests presenting as AI fetchers were refused — 37% of them.
By Vijay Vasu, Founder, Indexable. Published September 9, 2026.
How we measured. Five AI user-agent strings were sent to each of the seven domains ranking for “enterprise technical seo” in the United States on 9 September 2026 — n = 35 fetches — alongside 40 fetches across four of our own URLs and one robots.txt per domain. The matrix was run twice and all 35 statuses reproduced; a separate process re-fetched three domains the same day and matched. On five of the seven we probed the domain entry point, not the exact ranking URL, so this describes domain-level edge policy. No rendering measurement was taken, so nothing here supports a JavaScript claim about any domain. Seven hand-picked domains, one country, one day — a case study, not a law.
- Thirteen of 35 requests presenting as AI fetchers were refused with HTTP 403 — 37% — while all seven robots.txt files permitted them (Indexable, 2026).
- Zero disallow directives for AI agents appeared across those seven robots.txt files, over nine agent tokens checked (Indexable, 2026). The refusal sits at the edge, where a robots.txt review cannot see it.
- Three of seven domains refused at least one AI fetcher, and two of seven refused all five (Indexable, 2026).
- A 403 to a claimed user-agent string is evidence about edge policy, not vendor intent — bot-verification systems check IP and signed headers, not the string.
- Our own estate served all 40 fetches with zero blocks and still scored 33 of 100 on our own agent-readiness scale (Indexable, 2026). Access and readiness are separate tests.
What is enterprise website operations in the age of AI?
Enterprise website operations is the work of keeping a large site correct, fast and legible for every system that consumes it. For two decades that meant two consumers: a person in a browser and a search crawler. A third arrived without asking, and it behaves like neither.
| Consumer | How it arrives | What it needs | What breaks it |
|---|---|---|---|
| Human in a browser | Renders the page, scrolls, decides | Layout, speed, persuasion | Slow loads, confusing structure |
| Search crawler | Fetches, queues, renders on a later pass | Crawlable links, indexable HTML, clean canonicals | Blocked paths, duplicate signals |
| Retrieval system or agent | Fetches over HTTP, runs no JavaScript, takes a passage | Content in the raw response, self-contained claims, machine-readable facts | Edge refusals, client-side rendering, page bloat |
The third row is the whole story. A retrieval system does not queue your page for a render pass and return tomorrow. It makes one request, takes what the response contains, and moves on. If that response is a 403, or an empty shell that fills in after JavaScript runs, the page is absent from the answer — and absent looks identical to “not good enough” in every report you run today.
This belongs to operations rather than to content. Nobody writes their way out of a firewall rule.
Can AI fetchers reach the pages Google ranks?
Frequently not. On 9 September 2026 we sent five AI user-agent strings to each of the seven domains ranking on page one for “enterprise technical seo” in the United States. Thirteen of those 35 requests were refused with HTTP 403, and 22 were served (Indexable, 2026).
The refusals were concentrated. Three of the seven domains refused at least one AI fetcher, and two of them — neilpatel.com and quora.com — refused all five (Indexable, 2026). One split the difference: clutch.co refused requests presenting as GPTBot, OAI-SearchBot and CCBot while serving those presenting as ClaudeBot and PerplexityBot (Indexable, 2026).
Now the attribution discipline, which matters more than the headline. A 403 returned to a claimed user-agent string is evidence about edge policy and nothing more. Bot-verification systems check source IP and signed headers, not the text a client puts in its user-agent field. The honest sentence is “this domain refused a request presenting as GPTBot on 9 September 2026”. The dishonest one is “this domain blocks OpenAI”, and it is the version that gets repeated back to you.
This is where the series thesis lands on operations. Fifteen of our own 18 pages in Google's top 10 were never cited by an AI engine (Indexable, 2026). Ranking has stopped predicting retrieval, and one mechanical reason is that the fetch never completed.
Why will a robots.txt audit not find that refusal?
Because robots.txt is a published request and the edge is the enforcement, and the two can disagree completely. We fetched all seven robots.txt files and checked nine AI agent tokens in each — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot, Google-Extended, anthropic-ai, Applebot-Extended and meta-externalagent. Across all seven files, zero disallow directives applied to any of them (Indexable, 2026).
Every one of those files said yes. Thirteen requests were refused anyway.
This is an ownership problem before a technical one. robots.txt is written by the SEO team and lives in the repository. The rules that returned those 403s live in a CDN or WAF console owned by security or platform engineering, and are frequently a shipped default nobody chose. A bot-management preset arriving with “block AI scrapers” switched on produces exactly this pattern, and no audit that reads files rather than making requests will see it.
The Common Crawl team put it plainly: if you are not in the crawl, you are not in the model (Common Crawl, 2026). Make the requests, record the statuses, and route any 403 to the team that owns the edge.
Find out which AI fetchers your own estate refuses
The free AI search audit runs the same fetch matrix described above against your domain, and reports every status alongside what your robots.txt actually permits.
What does serving every fetcher actually prove?
Less than you would hope. Our own estate served all 40 fetches we sent it, with zero blocks — and scored 33 of 100 on our own agent-readiness scale (Indexable, 2026). Being reachable is the entry requirement, not the achievement.
That 33 breaks into four layers: discoverability 67, content accessibility 0, bot access control 33, and protocol discovery 33 (Indexable, 2026). Discoverability scores well because robots.txt and sitemap.xml are correct, and loses a third because no Link: headers are served. Content accessibility is zero because no markdown variant of any page is offered to an agent that asks. Protocol discovery is partial: an API catalogue and an agent card are published at well-known paths, while an MCP server card and OAuth discovery are not.
Two honesty notes belong with that number. The 0-to-100 figure is our own scale, not a vendor dashboard score, and must never be presented as one. And every signal was measured by probe, with a deliberately non-existent path as a negative control — it returned 404, confirming the 200s were real files (Indexable, 2026).
Our llms.txt sits in the same category. It is present, structurally valid, organised by task rather than by page hierarchy, and roughly 3,063 estimated tokens (Indexable, 2026). It carries no per-page token counts, so an agent cannot budget a fetch before making it. Report llms.txt as information rather than as a win: it is requested in about 0.1% of crawler visits and shows no correlation with citation frequency (Scauri, 2026).
Never assert a rendering gap you have not measured
Our house rule has one clause: no rendering-gap claim without a measured raw-versus-rendered word count. The 9 September pull took no rendering measurement, so it licenses no JavaScript claim about any of those seven domains, including the two that refused every fetcher.
The measurement is not difficult, which is what makes guessing inexcusable. Fetch the page over HTTP and count the words in the raw response. Render the same page in a headless browser and count the words in the DOM. The ratio of the first to the second is the share an AI fetcher can see, because most retrieval systems read the raw response and never wait for a render pass.
The bands to score against are these: raw text at 90% or more of rendered is low risk, 60% to 90% is medium, 30% to 60% is high, and below 30% is critical. A second check counts anchor tags in each version, because links appearing only after JavaScript are links a raw-HTML fetcher never follows.
Be strict here for diagnostic reasons. An edge refusal and a rendering gap produce the same symptom — your page is missing from answers — and need different owners and different fixes. Guess wrong and you spend a quarter re-platforming a front end when the problem was one checkbox in a bot-management console.
What does token economics change about how you build a page?
Token count becomes a page-design constraint alongside file size and load time. An agent inside a context window makes a budget decision about your page before reading the content, and an oversized document is silently truncated or passed over for a shorter one answering the same question.
The working targets come from Addy Osmani's AEO framework: under roughly 15,000 tokens for a quick-start page, under 25,000 for a standard page, and 30,000 as the ceiling before chunking by topic (Osmani, 2026). Most marketing pages sit inside that. Documentation sets, pricing matrices and long pillar pages routinely do not.
The retrieval evidence points the same way from a different angle. The citation window sits between 500 and 2,000 words (AirOps, 2026), and pages over 3,000 words surface only about 13% of their content to grounding systems (Scauri, 2026). Roughly 41% of AI citations come from the first third of a page (AirOps, 2026), so the back half of a very long page is largely writing for nobody.
Apply this at the template level, not the article level. Set a token budget per page type, expose the count in metadata so an agent can plan its read, and split anything over budget.
Structured data is a contract, not a decoration
Structured data is the cheapest asset website operations owns: a machine-readable statement of what the page asserts, and a contract between your claims and the systems quoting them. It is a retrieval lever rather than a ranking lever, and a modest one.
Be precise about the size of it. JSON-LD is associated with a citation lift of 6.5 percentage points, and tables plus lists together add 2.9 (AirOps, 2026). Those are the measured improvements, worth a template change.
Two further figures circulate constantly and are routinely misread. Pages carrying FAQPage markup are cited at a rate of 45.6%, and pages carrying BreadcrumbList markup at 46.2% (AirOps, 2026). Those are citation rates — the share of such pages that received a citation — not the gain from adding the markup. Restating a rate as a lift turns a 6.5-point decision into an apparent 45-point one, and it survives every quality review because the number is real and only the unit is wrong. We made that error in our own material before catching it.
The contract extends past schema. A verified source pack publishes operational facts with their constraints attached, including what the product does not do, because a pack listing only capabilities reads as marketing rather than truth (Forrester, 2026). Serve it at a stable well-known path and reference it from Organization schema.
How do you run website operations for three consumers?
Seven steps. The first two need a terminal and no budget.
- Step 1 — fetch your own pages as an AI fetcher. Start by sending the five common AI user-agent strings to your top 20 URLs and recording each HTTP status. Anything other than a 200 is your finding.
- Step 2 — read the edge, not the file. You should compare those statuses against what robots.txt permits. Where they disagree, the ticket goes to whoever owns the CDN or WAF.
- Step 3 — measure raw against rendered before blaming JavaScript. Count words in the HTTP response and in the rendered DOM, then apply the bands above. You can do this with curl and a headless browser.
- Step 4 — implement structured data as hygiene. Ship JSON-LD on every template and add tables and lists where the content supports them.
- Step 5 — set a token budget per page type. Fix the ceiling, expose the count in metadata, and split anything over budget rather than trimming it.
- Step 6 — publish a verified source pack with exclusions. List what you do, what you do not do, and the date each fact was verified.
- Step 7 — schedule the re-run. Next, put the fetch matrix on a 14-day cadence. A 403 today may be a 200 next week.
Fewer than four of these seven in place means you are operating a website for two consumers out of three. Our own score, on the day we ran it, was three.
In summary
Website operations acquired a third consumer, and most estates have never tested for it. The test is cheap: send five user-agent strings at your own top pages and read the statuses. On the seven domains Google ranks for a core enterprise term, 13 of 35 such requests were refused while every robots.txt file said they were welcome (Indexable, 2026). Run step 1 against your own domain this week and find out whether the systems answering your buyers' questions can reach you at all.
The Three-Consumer Check
- Have you sent the five common AI user-agent strings at your top 20 URLs and recorded every HTTP status, in the last 14 days?
- Have you compared those statuses against what your own robots.txt permits, token by token?
- (Pause point — if item 1 returns any non-200 while item 2 says "allowed", stop. That is an edge ticket, and it is not the SEO team's. Route it before continuing.)
- Do you have a measured raw-versus-rendered word count for your top templates, taken in the last quarter?
- Does every template ship JSON-LD, and do your data-carrying pages ship a table and a list?
- Do you know the token count of your heaviest page type, and is it exposed in metadata?
- Do you publish operational facts with their exclusions attached — what the product does not do — at a stable, machine-readable path?
Scoring — count the yeses:
- 6–7 yeses — Agent-legible. You are operating for three consumers. Move to cadence: re-run item 1 every 14 days.
- 4–5 yeses — Reachable but not ready. Access is probably fine; the machine-readable layer is thin. This is where we scored.
- 2–3 yeses — Untested. You are managing a website for two consumers and hoping about the third.
- 0–1 yeses — Blind. Any absence from AI answers is currently unattributable: you cannot tell an edge refusal from a rendering gap from a content problem.
We scored Reachable but not ready — three of seven the first time we ran this on ourselves. That is the honest baseline.
Frequently asked questions
Does refusing AI crawlers protect our content?
It protects the content from being retrieved, which may not be what was intended. Training crawlers and retrieval fetchers are different classes deserving separate decisions: refusing a training crawler is a licensing position, while refusing a retrieval fetcher removes you from live AI answers in your own category. Across the seven domains fetched on 9 September 2026, 13 of 35 requests were refused (Indexable, 2026) — and in every case robots.txt said the opposite, which suggests defaults rather than decisions.
How do we know whether our site has a rendering gap?
By measuring it, never by inferring it from the framework in use. Count the words in the raw HTTP response and the words in the rendered DOM, then take the ratio: 90% or more is low risk, 60% to 90% is medium, 30% to 60% is high, and below 30% is critical.
Is llms.txt worth implementing?
Implement it, then stop talking about it. llms.txt is requested in about 0.1% of crawler visits and shows no correlation with citation frequency (Scauri, 2026), which makes it a hygiene item rather than a lever. If you publish one, organise it by task, describe what an agent will find at each link, and include per-page token counts — ours has the first two and lacks the third (Indexable, 2026).
Vijay Vasu is the founder of Indexable. Crawler-access figures were pulled by direct HTTPS fetch on 9 September 2026 and re-run the same day; AI-citation figures come from a join dated 7 September 2026. Verified September 9, 2026.
Related reading
- AI-native websites — building for agents, not only humans.
- Enterprise SEO, in the Age of AI — why ranking and retrieval came apart.
- Technical SEO agents — the crawl and render layer beneath it.
Run the fetch matrix against your own domain
We will send the common AI user-agent strings at your top pages, compare every status against your robots.txt, and send you the disagreements.