Which AI Crawler Analytics Tools Show Whether ChatGPT and Perplexity Visit Your Website in 2026?

The most reliable way to tell whether ChatGPT or Perplexity visits your website is to combine request logs with a crawler identity table and an AI citation tracker. Server logs show a request; Cloudflare Logpush can add edge metadata; an AI visibility tool shows whether pages appear in generated answers. No single dashboard proves all three. Cloudflare's 2025 measurements show why: GPTBot rose from 2.2% of AI and search crawler traffic in May 2024 to 7.7% in May 2025, while ChatGPT-User requests rose 2,825% to a 1.3% share. Treat a bot request as a fetching signal, not a sale or citation.

For a small site, the practical stack is a web-server access log, Cloudflare or another edge log source, GA4 for referrals, and a prompt-monitoring tool for citations. The GEO measurement workflow covers the outcome layer; this report focuses on identifying the requests that happen before an AI system can cite a page. The GEO Cold Start playbook ($39.90, with AI citation tactics and a Claude Code skill) is useful when the log report shows that important pages are not being fetched or cited.

What did AI crawler traffic do from May 2024 to May 2025?

AI and search crawler activity increased, but the growth was uneven. Cloudflare's May 2025 comparison found an 18% increase from May 2024 to May 2025. Including newly added Cloudflare customers, the increase was 48%. April 2025 peaked at 32% above the May 2024 baseline. These figures describe Cloudflare's network, not every website, so treat them as market direction.

The operator mix changed faster than the total. GPTBot requests increased 305%, moving from 2.2% to 7.7% between May 2024 and May 2025. ChatGPT-User requests increased 2,825% and reached 1.3%. PerplexityBot recorded a 157,490% increase in raw requests, yet held only a 0.2% share. The large Perplexity percentage reflects a small base; it does not mean Perplexity became the largest crawler.

A second Cloudflare study found Googlebot at 39% of AI and search crawler traffic in July 2025. GPTBot rose from 4.7% in July 2024 to 11.7%, while ClaudeBot rose from 6.0% to 9.9%. Meta-ExternalAgent increased from 0.9% to 7.5%; Bytespider fell from 14.1% to 2.4%. Track counts and shares: counts reveal workload, while shares reveal relative demand.

All figures come from Cloudflare's 2025 crawler analysis and Cloudflare's July 2025 crawler report. Record the timestamp, path, status, bytes, and verification result before aggregation.

Which crawler signals should a small website separate?

Separate four request classes: training crawlers, search crawlers, user-triggered fetchers, and ordinary search crawlers. OpenAI identifies GPTBot for generative AI model training and OAI-SearchBot for search. Perplexity describes PerplexityBot as surfacing and linking websites in Perplexity results. ChatGPT-User and Claude-User fetch pages after a person asks an assistant to inspect a URL. Googlebot and Bingbot support discovery without proving that an AI answer used the page.

The robots.txt token and the HTTP user-agent are related but not identical evidence. Google states that Google-Extended controls whether content may be used for future Gemini training and grounding, while Google-Extended does not affect inclusion in Google Search. The Google crawler documentation is the reference for that distinction. OpenAI's crawler documentation likewise separates GPTBot from search and user-request agents. Record declared policy and observed requests independently.

Do not label every user-agent containing an AI brand as verified. A spoofed agent can imitate GPTBot, ClaudeBot, or PerplexityBot. Prefer published IP-range verification and mark matches as “claimed,” “verified,” or “unknown.” A 200 response from a claimed bot is weaker evidence than a verified bot fetching canonical HTML. A useful report also records whether a request reached the canonical HTML rather than a redirect or asset. This classification prevents a crawler's technical noise from being mistaken for content discovery.

Which tool stack should you use in 2026?

Use the least expensive layer that answers the question. For “Did a crawler request this URL?”, use the origin or CDN log. For “Was it verified?”, use edge bot fields. Cloudflare's dataset documents ClientRequestUserAgent, BotScoreSrc, and BotTags; the latter fields depend on Cloudflare Bot Management. The Cloudflare HTTP requests documentation lists fields and plan limitations.

For “Did an AI answer cite my page?”, use a prompt-monitoring or AI visibility tool and maintain a fixed prompt set. The internal linking structure matters because the answer layer cannot replace logs: citation trackers sample prompts, while logs record requests whether or not a page appears in an answer. Logs cannot replace citation monitoring because a training crawler may fetch a page without sending a referral or producing a visible citation.

1. Access logs: best for URL, time, status, latency, bytes, and user-agent evidence.
2. Cloudflare Logpush or equivalent: best for edge-level aggregation, bot classification fields, and request exports.
3. GA4: best for measurable referrals and engaged sessions; it will not expose crawler requests that never execute analytics JavaScript.
4. AI visibility tracker: best for prompt coverage, cited URLs, share of voice, and competitor comparisons.

Logs have high request coverage but low interpretation. Edge analytics add bot context but can hide plan-dependent fields. GA4 measures human outcomes but misses non-browser fetching. Visibility tools measure answer presence but sample the market. The small-site minimum is logs plus a prompt set; add edge analytics when volume or spoofing makes raw logs hard to manage.

How should you build a 14-day crawler report?

Export one row per request for 14 consecutive days. Keep timestamp, path, status, response time, bytes, user-agent, referrer, and verification result. Normalize paths so tracking parameters do not split an article. Exclude CSS, JavaScript, fonts, and images from the article-fetch report. Keep a second asset count because incomplete resource fetches can affect what a crawler sees.

Use this calculation for each crawler class: Signal Quality Score = verified HTML fetches x successful status rate x canonical-page rate. Express the last two terms as decimals. For example, 40 verified HTML fetches, a 95% 2xx rate, and an 80% canonical-page rate produce 30.4 quality-weighted fetches. This is not a search-engine metric; it prevents failed or non-canonical requests from looking like strong coverage.

• Count verified and unverified requests separately for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot.
• Track the number of unique canonical article URLs fetched by each class.
• Track 2xx, 3xx, 4xx, and 5xx status rates, plus median response time.
• Track AI citations for the same URLs across a fixed set of at least 20 prompts.
• Track AI referral sessions, engaged sessions, email signups, and sales with UTM or referrer data.

Compare the first seven days with the second seven days. Rising verified fetches with flat citations suggests a content interpretation or prompt coverage problem. Rising citations with flat referrals suggests that the page is evidence but is not converting the visit. A falling 2xx rate points to delivery trouble. If all metrics are flat, publish one evidence-rich page in the highest-demand cluster, confirm that the canonical URL is crawlable, and rerun the same report.

How do you interpret crawler activity without overstating demand?

Use cautious language. “GPTBot fetched three URLs” is an observed fact. “ChatGPT is sending customers” requires referral evidence. “The page was cited” requires an answer observation with the prompt, date, engine, and URL. A large growth percentage can coexist with a modest operational footprint.

Use a two-column interpretation: identity confidence and business relevance. Identity confidence rises from user-agent match to published IP verification. Business relevance rises from HTML fetch to citation, referral, signup, and sale. Keep these scores separate so a high-confidence training request does not receive the same value as a user-triggered fetch that led to a conversion.

Review policy before changing robots.txt. Perplexity says PerplexityBot indexes and links pages in search, not foundation-model training. Google says Google-Extended is a separate control for Gemini training and grounding. OpenAI separates GPTBot, OAI-SearchBot, and ChatGPT-User. Choose access based on licensing, privacy, server cost, and visibility. Log the policy change, then compare the next 14-day report with the baseline.

Frequently Asked Questions

How do I verify that an AI crawler really visited my site?

Match the user-agent to the operator's published documentation, check the request against published IP ranges when available, and confirm that the request fetched an HTML page with a normal status code. Mark user-agent-only matches as unverified.

When should I start tracking AI crawler activity?

Start before a content launch or robots.txt change, and collect 14 days of baseline data.

Which tool is better: server logs or an AI visibility platform?

Server logs answer whether a request happened. An AI visibility platform answers whether a monitored prompt produced a citation. Use both when citation is the business goal; choose logs alone for crawlability.

How do I measure the business return from crawler access?

Connect cited URLs to AI referrals, engaged sessions, email signups, assisted conversions, and sales. Report crawler fetches as an upstream signal and revenue as a downstream result. Do not assign revenue to a bot request without a human-session path.

Is AI crawler tracking worth the setup for a small website?

It is worth the setup when a site publishes regularly or needs evidence for access-policy decisions. For the distribution layer after measurement, the bilingual Reddit Marketing Playbook provides a separate organic acquisition workflow; evaluate it by tracked visits and signups, not crawler volume.