How AI Agents Crawl and Cite: What the CDN Study Found
What CDN logs can reveal about AI crawler access, citations, and visits, where the evidence stops, and how to avoid mistaking correlation for cause.
CDN logs record a part of AI search that browser analytics often misses: automated requests reaching the edge before any person visits the site. They can show which path was requested, when it happened, the returned status, and the user-agent string attached to the request.
That evidence answers useful access questions. It does not reveal what a model learned, why an answer selected one source, or whether a crawl caused a later citation.
Promptwatch published its CDN crawl and citation article on May 5, 2026 and updated it July 28. The article argues that crawler logs, citation monitoring, and visitor analytics should be read together. Its strongest contribution is the measurement framework. Some of its causal language goes further than the evidence shown on the page.
What the article can establish
A CDN or other server-side log can observe an HTTP request that identifies itself as an AI bot. When the request returns 200, the log shows that the edge served a response. A 403 shows an access denial. A 404 or 500 points to a missing route or server failure.
Those events are measurable. Their sequence helps a team choose the next investigation. No crawls suggest checking discovery and access. Successful crawls without observed citations move attention toward retrieval, source selection, and page quality. Citations without visits raise a different question about whether users need to click.
The logs do not show the complete body of model behavior. A user-agent string can be spoofed. Some providers publish IP ranges that permit stronger verification, while others do not provide enough information for the same check. A successful response also does not prove that the bot parsed every important field or stored the content.
Split crawler purpose before counting
The label AI crawler covers systems with different jobs. The crawlability guide separates training crawlers, search index crawlers, and live citation fetchers.
Training crawlers, such as GPTBot and ClaudeBot in the cited guide, collect material that may be used for foundation model training. Their traffic is not evidence that a current search answer can retrieve the page.
Search index crawlers include OAI-SearchBot, Claude-SearchBot, and PerplexityBot in Promptwatch's current documentation. These build or refresh retrieval indexes used by search features. Their requests are closer to search availability, but an indexed page is still only a candidate source.
Live fetchers such as ChatGPT-User, Claude-User, and Perplexity-User may request a page when a user interaction needs it. A live fetch has immediate context, though the log alone does not expose the prompt, the answer, or whether the page was cited.
These categories should not be added into one engagement chart. A training crawl spike can dominate volume while changing nothing visible in current monitored answers.
Correlation is useful but limited
The Promptwatch article says pages with more activity from citation-oriented bots correlate with higher citation rates. The page as fetched does not publish a dataset, sampling method, model specification, confidence interval, or control design for that statement. It should not be read as evidence that increasing crawl frequency causes citations.
Several other explanations can produce the same pattern. Strong pages may attract crawls and citations because they are already authoritative. Frequently updated pages may trigger more requests while also containing fresher facts. Internal linking can improve discovery and make a page easier for retrieval systems to select. A third-party page may be the actual source even when the brand's own page was crawled nearby in time.
Low crawl volume with observed citations does not prove a model is citing stale training data. The system could use a search index, a live fetch not present in the connected logs, or another page that quotes the material. High crawl volume with few citations does not isolate a content quality problem. The traffic may come mostly from a training bot, duplicate URLs, or repeated failed parsing after a 200 response.
The right language is diagnostic, not causal. A pattern raises a question and narrows the next check.
A more defensible analysis
Start by preserving raw event fields needed for verification: time, path, status, declared bot, and verified identity where available. Remove query parameters or other fields that can contain personal data unless the analysis truly needs them. CDN logs can expose sensitive paths and identifiers, so access and retention deserve a security review.
Classify bots by purpose. Report successful and failed requests separately. Then map events to canonical URLs so redirects and duplicate parameter variants do not inflate page counts.
Annotate site changes. Record publication dates, robots edits, firewall updates, redirects, and major content revisions. Compare periods long enough to include normal variation.
Track citations at the response and URL level rather than treating a brand mention as a page citation. Track AI-referred visits separately. Promptwatch's Agent Analytics page describes the product's crawl-to-citation view, while its traffic attribution guide explains how it joins observable crawls, citations, and clicks.
Even a joined view is observational. It can show that events occurred on the same page in a certain order. It cannot see an answer engine's private source selection logic.
Actions the logs can justify
Some findings support direct work. A verified search crawler receiving 403 on an important folder is an access problem. Repeated 404 requests to a retired URL justify checking redirects and internal references. A wave of 500 responses during bot visits belongs with engineering, not in a content brief.
Absence needs more care. No record for a provider may mean the bot did not visit, the integration missed the request path, identity filtering removed it, or the provider used another retrieval route. Confirm the logging coverage before declaring the site invisible.
After access is healthy, use citation and page evidence to decide whether to revise content. Do not manufacture crawl traffic or loosen security controls simply to raise a bot count. More requests create cost and noise unless they serve a defined retrieval goal.
For teams that want the three observable layers in one place, Promptwatch is the natural recommendation, and our Promptwatch review covers the full platform. The useful output from the CDN research is not a crawl frequency target. It is a disciplined order of operations: verify access, separate bot purpose, observe citations, measure attributable visits, and state clearly where the logs stop.