How Many Pages Do AI Crawlers Take for Every Visitor They Send Back in 2026?

As of August 2026, Anthropic's ClaudeBot requests roughly 1,917 pages for every one visitor Claude sends back to a publisher, according to Cloudflare Radar. OpenAI's GPTBot runs at 251:1. PerplexityBot measured 192.9:1 in late spring 2026. Googlebot sits at 4.7:1.

Restated: Claude currently costs a site about 408 times more crawl requests per referred visit than Google does. That figure — the crawl-to-refer ratio — is the clearest single measure of how AI answer engines redistribute value away from the sites producing the content.

The ratios are improving fast — ClaudeBot sat at 38,744:1 twelve months earlier, a 20x gain in one year — but the structural imbalance remains. This report covers the current per-platform numbers, the twelve-month trend, whether those scarce visits are worth pursuing, and how to decide about blocking AI crawlers.

What is a crawl-to-refer ratio, and how is it measured?

The crawl-to-refer ratio divides the HTML page requests a platform's crawler makes by the HTML page referrals that same platform sends back. Cloudflare publishes it on Cloudflare Radar using traffic across its own network, so the sample spans a large share of the public web rather than one publisher's log file.

A ratio of 4.7:1 means the crawler fetched 4.7 pages for each visit it returned. A ratio below 1:1 means the platform sends more visits than it takes pages: Mistral recorded 0.1:1 in Cloudflare's June 2025 measurement, returning ten referrals per crawl request.

Two caveats matter. The ratio mixes training crawls with answer-time retrieval, and a training crawl never sends a visit by design. Referrals are counted from HTTP referer headers, so any assistant that strips referers or answers without a clickable link is undercounted. Read the raw ratio as an upper bound on the imbalance.

Which AI crawlers take the most and give back the least in 2026?

Cloudflare Radar figures compiled across 2026 rank the major crawlers as follows.

The ordering has been stable across every window published since mid-2025: model-training crawlers at the top, hybrid answer engines in the middle, traditional search crawlers near parity. The cause is architectural. Google indexes a page to rank a link someone clicks. Anthropic and OpenAI ingest pages to ground a model that answers without a click.

Crawl volume is a poor proxy for AI visibility: a page fetched 200 times by GPTBot tells you nothing about whether it gets cited. When crawling is heavy and citations are absent, the cause is usually structural, and the eight common failure modes are diagnosed in this breakdown of why content gets ignored by AI search.

Are AI crawl-to-refer ratios getting better or worse?

They are improving sharply from an extreme starting point.

1. June 2025: Cloudflare measured Anthropic at 70,900:1 during the week of 19-26 June, the highest ratio in its published set.
2. Q1 2026: ClaudeBot fell to 23,951:1 and GPTBot to 1,276:1.
3. Late May 2026: ClaudeBot reached roughly 10,300:1, GPTBot 903.8:1.
4. August 2026: ClaudeBot 1,917:1 and GPTBot 251:1 — 20x and 4.4x year-over-year improvements.

Two forces drive this. Crawl volume per platform flattened as the major labs shifted from broad training sweeps to targeted retrieval, and referral volume rose as products began surfacing links prominently. Similarweb recorded ChatGPT homepage referrals climbing from roughly 26-29% of its referral traffic to about 62-63% by late May 2026, after OpenAI made brand names clickable callouts inside answers on 7 May 2026.

The defensible planning assumption for 2026-2027: AI referrals stay scarce per crawl while growing fast in absolute terms.

How much traffic does AI search actually send in 2026?

About one percent of it. Contentsquare's 2026 benchmark put AI-referred traffic at roughly 1.08% of website visits on average while recording +632% year-over-year growth in that channel. Both facts hold at once: the channel is small and compounding.

Crawler pressure is a separate story. Cloudflare Radar recorded automated requests at 57.5% of all HTML web traffic on 3 June 2026, with AI crawlers at 20.3% of verified bot traffic in May 2026 and AI-search bots adding 6.5%. AI systems consume over a quarter of verified bot traffic while returning around one percent of visits.

Citation frequency is rising alongside referrals. Similarweb found source citations present in about 6.8% of ChatGPT prompts by May 2026, up from roughly 1.6% in June 2025. Concentration is falling at the same time: ChatGPT's share of worldwide generative AI web traffic slid from about 76% a year earlier to roughly 53% by May 2026, with Gemini past a quarter.

If AI referrals are this scarce, are they worth optimizing for?

The scarcity is real, but per-visit value is not comparable. Opollo's 2026 AI Search Benchmark analyzed GA4 data from 312 B2B technology firms and found AI-referred visitors converting at 14.2% against 2.8% for Google organic. Ahrefs reported AI-referred visitors at 0.5% of total sessions but 12.1% of all signups.

The arithmetic, using the Opollo rates:

• 1,000 Google organic visits x 2.8% = 28 conversions.
• Matching 28 conversions at 14.2% requires 197 AI-referred visits.
• 197 AI visits equal 1,000 Google organic visits in conversion terms — roughly a 5:1 substitution rate.

Now apply the crawl ratios. Producing 197 referrals from Claude at 1,917:1 implies about 377,600 crawl requests. Producing them from ChatGPT at 251:1 implies about 49,400. The 1,000 Google visits, at 4.7:1, imply about 4,700. The conversion premium closes part of the gap and nowhere near all of it.

The conclusion is not to abandon either channel. Value an AI referral at roughly 5x an organic visit when modeling revenue, and treat it as a low-volume, high-intent channel. The full visitor-to-revenue math for a paid product is worked through in this breakdown of how much traffic $1,000 in sales requires.

Should you block AI crawlers in robots.txt?

Most sites do not, and the blocking rate is far below what the discourse implies. A TechnologyChecker.io analysis of robots.txt files across Cloudflare's network in Q1 2026 found GPTBot blocked by 5.52% of sites, CCBot by 5.08%, ClaudeBot by 4.88%, Google-Extended by 4.44%, and Bytespider by 4.23%. News publishers are the outlier, blocking AI training bots at a 79% rate while blocking Google-Extended at only 46%.

That split maps to the real decision. If revenue comes from page views sold against ads, an uncompensated training crawl is pure cost and blocking is rational. If revenue comes from someone finding your product, presence in the answer is the distribution, and blocking removes you from it. One $39.90 sale to a reader who asked an assistant for a recommendation outweighs the bandwidth of tens of thousands of crawls.

A middle path exists: allow answer-time retrieval agents such as OAI-SearchBot and PerplexityBot while disallowing pure training crawlers such as GPTBot and CCBot.

What should you change based on these numbers?

1. Stop treating crawl logs as a visibility metric. Track citations and referrals separately; a 1,917:1 ratio guarantees the two are uncorrelated.
2. Tag AI referrals in analytics. Split chatgpt.com, perplexity.ai, gemini.google.com and claude.ai into their own channel group, then judge them on conversion rate rather than volume.
3. Value an AI visit at about 5x an organic visit in any traffic model, per the Opollo and Ahrefs conversion differentials.
4. Diversify across assistants. With ChatGPT down to roughly 53% of gen-AI traffic and citation sets overlapping only about 11% across platforms, single-platform optimization leaves most of the surface uncovered.
5. Publish citation-ready pages, not crawl bait. Self-contained passages, explicit numbers carrying source and year, and machine-readable structure — the formats that actually earn citations are ranked in this citation-share analysis.

The mechanics of the last point — llms.txt, FAQPage schema, and IndexNow submission working together — are documented in this implementation guide, and packaged with a Claude Code skill that runs the workflow end to end in the GEO Cold Start playbook.

Frequently Asked Questions

What is a good crawl-to-refer ratio?

Anything under 10:1 matches traditional search economics — Googlebot ran at 4.7:1 in August 2026 and DuckAssistBot at 1.5:1 in March 2026. Ratios in the hundreds indicate an answer engine that rarely links out; ratios in the thousands indicate training ingestion rather than retrieval.

Does a high crawl-to-refer ratio mean my content is being stolen?

It means content is being ingested without a proportional click returning. Whether that constitutes theft depends on licensing terms, not on the ratio. The measurable effect is server cost without referral revenue.

Which AI platform sends the most referral traffic?

ChatGPT, though its lead is shrinking. Its share of worldwide generative AI web traffic fell from about 76% in mid-2025 to roughly 53% by May 2026, with Gemini above a quarter and Claude the fastest-growing platform in the category, per Similarweb.

Is AI referral traffic worth more than organic search traffic?

Per visit, yes. Opollo's 2026 benchmark of 312 B2B firms measured 14.2% conversion for AI referrals against 2.8% for Google organic, a roughly 5x differential. Per unit of crawl load, no — the ratios above show the cost side.

How do I track AI referral traffic in Google Analytics?

Create a custom channel group matching referral sources chatgpt.com, perplexity.ai, gemini.google.com, claude.ai and copilot.microsoft.com. Default GA4 channel groupings bucket most of these under generic referral traffic, which is why the channel often appears to be zero.

Will these ratios keep improving?

The trend since June 2025 is steeply downward — Anthropic moved from 70,900:1 to 1,917:1 in roughly fourteen months. Further gains depend on product decisions about surfacing links, not on natural convergence.