The Crawl-to-Citation Path: Following One Page From Bot Request to Cited Source
A page has to be fetched, understood, and chosen before it is cited. Here is how to trace each step and find where yours stops.
When a page is not cited, teams usually guess. The content must be weak, or the topic too competitive, or the engine biased. Sometimes that is true. More often the page fell out of the process earlier than anyone checked.
A citation is the last step of a short chain. The page has to be discovered, fetched successfully, read, judged relevant to a question, and chosen as a source. If you can see each link of the chain for one page, diagnosis becomes a matter of finding where the chain breaks.
The five checkpoints
1. Discovery. The bot has to learn the URL exists. Sitemaps, internal links, and external links all help. A page with no internal links and no sitemap entry may never be requested.
2. Fetch. The bot makes a request, and your server answers. A 200 is the goal. A redirect chain, a timeout, a challenge page, or a block returns something else.
3. Content. The response has to contain the content in a form the bot can use. Pages that depend on scripts to render their text may return an empty shell to a bot.
4. Relevance. The text has to answer a question that engines are asked. This is a content question, and it comes last on purpose. Many teams start here and spend weeks rewriting pages that were never fetched.
5. Selection. Among all relevant pages, the engine picks a few to cite. This is where competition lives.
Where each checkpoint shows up in data
Checkpoints one to three are visible in server or CDN logs. Checkpoint five is visible in citation data. Checkpoint four sits in between, and you infer it from the two sides.
Promptwatch's Agent Analytics ties these together under the name crawl-to-citation path. It reports real-time, IP-verified crawler logs, so requests with spoofed user agents are removed, and it reports a citation rate per page. It also tracks errors and gives failure explanations. Bots it tracks include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, and MistralAI-User.
Reading a page through the path
Pick a page you expect to be cited and ask four questions in order.
Did any bot fetch it in the last 30 days? If not, you are at checkpoint one or two. Check the sitemap, the internal links, and robots.txt. Check whether a firewall is rejecting bots.
Did the fetches succeed? If the logs show errors, read the explanations. A pattern of the same error on the same path is usually a configuration fix.
Is the page rendered in a form the bot can read? View the page the way a bot would, without running scripts, and confirm the main text is present.
Is it cited? If it is fetched, readable, and still not cited, you are at relevance or selection. Pin it with Page Tracker, pin the competitor page that is cited, and compare.
The order is the point. Do not rewrite a page until you have confirmed it is fetched.
What a healthy path looks like
For a page that works, you will usually see requests from the search and user-fetch bots, successful responses, and a citation rate that is not zero. Different bots have different jobs. A search bot request suggests indexing. A user-fetch bot request suggests a person's question led the engine to read your page. Both are good signs.
Treat the specific numbers as relative. A citation rate of a few percent might be excellent for a narrow topic and poor for a broad one. Compare pages inside your own site, and compare against competitor pages you pin.
Using the path to prioritize
Make a table with four columns: page, fetched, succeeded, cited. Fill it for your top 30 pages. Four patterns will appear:
- Not fetched: fix discovery and access.
- Fetched with errors: fix the errors.
- Fetched and succeeded, not cited: improve the content or build outside mentions.
- Cited: protect and refresh.
Each pattern maps to a different team. Access and errors go to engineering. Content goes to editorial. Outside mentions go to PR. A clear table reduces arguments about who owns the problem.
Linking to traffic
The path can be extended one more step: from citation to visit. Visitor analytics in Promptwatch attribute AI-referred traffic and conversions through a lightweight script, a GTM template, or the WordPress plugin. A page that is cited but sends no visits may be answering the question entirely inside the AI response, which suggests changing what the page offers beyond the answer.
The 360 Insights view joins crawler logs, citation clicks, and AI-referred visits per page, which is the same chain in a single table.
Plan notes
Crawler logs start on the Professional plan. Essential has none, and Explore is limited to ChatGPT prompts. If checkpoints one to three are your main concern, plan for Professional or above. Agency plans include crawler logs on every tier.
Automating the weekly check
The weekly Discovery agent on the Action Board investigates your data and proposes up to five evidence-backed actions. A fetch error pattern or a page that dropped out of crawl is the kind of finding it can surface. Scheduled Chats can deliver a summary in the app or Slack, so the Monday reading is waiting for you.
The recommendation
When a page is not cited, trace the chain before you touch the copy. Promptwatch gives you the path in one view, with IP-verified crawler requests, error explanations, and a citation rate per page. Pick your five most important uncited pages and walk each through the four questions this week.