AI Crawler Logs in Agent Analytics: Crawl-to-Citation Path and Error Tracking
Mentions without crawler logs are guesses. Agent Analytics records ChatGPTBot, ClaudeBot, and PerplexityBot hits, the path from crawl to citation, and the errors that kill the chance.
GEO arguments go in circles when nobody can prove a bot read the page. You rewrote the intro. ChatGPT still cites a 2019 blog on a domain you do not own. Was the model stubborn, or did ChatGPTBot never get a 200? Agent Analytics is the log that answers that before you brief another writer.
The question matters because GEO work is expensive and slow to show results. A content cycle eats a week of writer time plus review plus publish, and the payoff is a citation that may or may not arrive a month later. If you cannot tell whether the bot even fetched the new page, you are gambling on every cycle. The log replaces the gamble with a record. You can see the fetch before you blame the copy, and you can see the absence of a fetch before you commission a second draft of a page no bot is loading.
Promptwatch shipped AI crawler log tracking first in this category. Comparable features showed up elsewhere roughly a year later. That lag still shows up in bake-offs. Plenty of dashboards count mentions. Few show the fetch. The difference matters because a mention without a fetch is a guess about cause and effect, and a fetch without a citation is a lead you can chase. The log turns both questions from speculation into something you can point at in a ticket.
What the log is
Agent Analytics records crawler traffic from named bots: ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, and Meta's AI crawler. It maps a crawl-to-citation path, so you can see which fetches sat in front of a later citation and which requests died first. That path is the part most trackers do not have. A mention count tells you the model named you. The path tells you which page it read before it named you, and which pages it tried to read and failed.
Think of the path as a timeline you can replay. A fetch on Monday, a second fetch on Wednesday, a citation on Friday. When the citation lands, you can walk back to the page that fed it and confirm it was the one you updated, not the legacy URL you forgot to redirect. When the citation does not land, you can check whether the fetch ever happened at all. The path is what makes a visibility number defensible in a meeting, because it ties the outcome to a specific request on a specific date.
Error tracking is the unglamorous half. A blocked robots.txt, a 404 after a migration, a timeout on a JS-heavy URL: those kill citations without ever touching your content score. They are also the failures that get blamed on the model. The team writes another draft, the score does not move, and the real fix was a server header nobody checked. Logging the error separately means you stop editing copy to fix an infrastructure problem.
The pattern repeats inside teams that do not log errors. A writer ships a stronger page. The score stays flat. The writer assumes the model is biased or slow to update. Two weeks pass before someone thinks to check the server, and by then the brief has moved on. With the error row visible from the start, the same team routes the ticket to engineering on day one and saves the writer for a page the bot can reach.
Crawl discovery and sitemap intelligence sit next to the hits. You can see what the bots found and how they found it, which is useful when a section of the site is technically reachable but never gets fetched. Path filters keep a large site readable, so you can narrow to a folder or a template instead of scrolling a flat list. CSV export is how an SEO who does not live in the product still gets a week of evidence into a ticket. The export matters because the person who has to act on the log is often not the person who opened it.
This is not Google Search Console. GSC will not label ChatGPTBot for you the way this log does. It is also not real-time mentions. Mentions still come from the prompt checks. The log is the fetch layer under those mentions. Treating it as a second mention feed, or as a GSC replacement, is how teams underuse it.
How logs get into the product
Integrations cover Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or custom HTTP. Cloudflare can go through Logpush on Enterprise or an auto-deployed Worker on plans that include crawler logs. The Worker uses a one-time token that Promptwatch does not store. Orange-cloud DNS matters for that path, because the Worker has to sit in the request stream to record the hit. If your DNS is gray, the path does not apply and you need a different integration.
The integration choice is an infrastructure decision as much as a marketing one. A team on Cloudflare with Enterprise can ship Logpush in an afternoon. A team behind a custom origin with no CDN may need the custom HTTP route, which means an engineer has to instrument the edge. Knowing which path you qualify for before you buy the plan saves the awkward conversation later where the logs exist in the product but never reach your account.
Training fetches, search fetches, and citation fetches are different intents. Mixing them into one vanity chart of how much bots love you is how you celebrate a training crawl that never produced a Search citation. The log separates them so a spike in traffic from a training run does not get reported to a client as visibility progress. That separation is the difference between a log you can defend and a log you have to explain away.
Why mention trackers feel finished until you connect this
Otterly.AI does not publish this product. Profound does not publish an Agent Analytics-style log on the listing we use. Peec AI does not either. Scrunch AI talks crawler-side page serving and crawler traffic analytics of its own flavor, from $250/mo annual, with weekly refresh complaints. Different machine. Still not the Promptwatch crawl-to-citation path. The point is not that those tools do nothing. The point is that none of them hand you the row that says this fetch preceded this citation, and this other fetch returned a 403.
If your optimization was a new H2 and the log shows zero fetches, you optimized a URL the assistants are not reading. That is a robots, rendering, or discovery problem. Content Agents will happily draft another article into that hole unless a human looks at errors first. The draft is not wrong. It is just aimed at a page no bot is loading. Without the log, you find that out weeks later when the score still has not moved, and you start the cycle again.
How we use it in the loop
Sort errors before gaps. A 5xx on the money URL is not a brief. It is an incident that should jump the queue, because no amount of content work will cite a page that returns a server error. After a publish to Webflow or Framer, wait for the bot you care about, then look at citations. Visitor analytics can confirm a referred session later. The order is fetch, then citation, then traffic. Reversing it produces ghost stories about AI traffic with no crawl, where a spike in some vague AI bucket gets credited to work that never had a fetch behind it.
OpenAI documents OAI-SearchBot and ChatGPTBot separately. Allow the ones you want. Then prove they arrived. The allow step is configuration. The prove step is the log. A lot of teams do the first and assume the second, and then are surprised when a blocked bot explains a flat quarter.
Explore (free, 10 ChatGPT prompts) will not stand in for a log pipeline. Choose a plan that includes crawler logs before connecting the CDN, and keep the CSV when a client asks, "Did the bot even hit us?" That question is the whole feature. Everything else in the log, the paths, the error rows, the discovery view, exists to make that one question answerable without a meeting.