AI Crawler Activity in Website Logs: Cloudflare AI Bots, Visibility Tools, Google AI Overviews Crawler, and Bing Copilot Crawler Analytics (2026)
There is no AI Overviews crawler and no Copilot crawler in your logs. What each AI surface actually fetches with, what Cloudflare shows, and how to turn a fetch into a fix.
Filter a week of access logs for anything that looks like AI and you will find GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. You will not find an "AI Overviews crawler" or a "Copilot crawler." Two of the biggest AI answer surfaces are fed by the oldest bots on the web, and that trips up more teams than any blocked user agent.
This site ranks tools by what happens after the data arrives. For crawler activity, the useful question is not "how many AI hits did we get?" It is which fetch failed, on which page, and what you change before the next answer gets generated. A crawler log that never leads to a fix is just a chart.
Which crawler feeds which surface
Your logs only show user agents, so the first job is mapping names to products.
| AI surface | What appears in your logs |
|---|---|
| ChatGPT | GPTBot, OAI-SearchBot, ChatGPT-User |
| Google AI Overviews and AI Mode | Googlebot |
| Gemini training and grounding | Nothing (Google-Extended is a robots.txt token) |
| Microsoft Copilot | Bingbot |
| Claude | ClaudeBot |
| Perplexity | PerplexityBot |
OpenAI documents GPTBot and OAI-SearchBot as separate robots, one for training and one for search. ChatGPT-User is a fetch a person triggered inside ChatGPT. Lumping all three into one "OpenAI" line hides the only distinction that matters for citations.
Google has no AI Overviews crawler
Google's crawler documentation (updated July 2026) says Googlebot's crawl preferences apply to Google Search, "including Discover and all Google Search features." AI Overviews and AI Mode are Search features. So the bot behind them is Googlebot, full stop. Block Googlebot to keep a page out of AI Overviews and you have left Search entirely.
Google-Extended causes the opposite confusion. It is a control token you put in robots.txt to opt out of Gemini training and grounding in Gemini Apps and Vertex AI. Google says it "doesn't have a separate HTTP request user agent string" and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." You can grep for it forever. It will never show up.
GoogleOther is a generic crawler that Google says doesn't affect any specific product. A GoogleOther spike is not a sign that AI Overviews took an interest in you.
Bing's Copilot crawler is Bingbot
Bing's webmaster guidelines say Bing and Copilot search experiences "rely on the same core crawling, indexing, and ranking foundation as traditional search." Blocking Bingbot in robots.txt is on Bing's list of mistakes. The controls for Copilot are page directives, not a separate bot. NOINDEX keeps a URL out of Bing, Copilot and grounding results. NOARCHIVE "prevents content from being used in Copilot responses and grounding results." NOCACHE limits Copilot to the URL, title and snippet. And remember that robots.txt controls crawling, not indexing.
Verify before you count anything
User agents can be spoofed. Google tells you to check reverse DNS (hostnames on googlebot.com) or its published IP ranges. Microsoft has a Verify Bingbot tool. A "Googlebot" surge from a random hosting range is a scraper wearing a costume, and it belongs in a block rule, not in your AI report.
What Cloudflare shows without extra tooling
If your site sits behind Cloudflare, AI Crawl Control (formerly AI Audit) is on every plan. Its Overview, Crawlers and Metrics tabs break requests down by crawler and operator (OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta), allowed versus unsuccessful requests, robots.txt violations, status codes, paths and hosts. You can allow or block each crawler, and pay per crawl is in private beta.
The free plan is thinner than people expect. Detection is by user-agent string only, the Metrics tab covers the past 24 hours, and referral counts are a paid-plan feature. Enterprise with Bot Management adds detection IDs and configurable timeframes, and the GraphQL Analytics API exposes the data if you want to pipe it elsewhere.
That is a real control surface. Cloudflare acts on access: it lets a bot in or keeps it out. What it doesn't do is tell you whether a fetch was followed by a citation, which prompt that page answers, or whether the visitor ChatGPT sent you went on to convert.
Where AI visibility tools fit
Most tools in our directory sample answers. They know ChatGPT cited a competitor's guide on Tuesday. They have no idea whether OAI-SearchBot fetched your own guide that week, or got a 404 on it. Logs have the opposite blind spot. You need both halves, joined on the same page and the same date, before you can say why something happened.
Joining the halves in Promptwatch
Promptwatch is the platform we point readers to for that join. Agent Analytics records real-time AI crawler logs for ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther and Meta's AI crawler, maps the crawl-to-citation path, and tracks errors separately so a 5xx doesn't get mistaken for a content problem. Logs come in from Cloudflare (Logpush on Enterprise, an auto-deployed Worker on other plans, with the record orange-cloud proxied), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or custom HTTP.
Be clear about one limit. Promptwatch's published crawler list doesn't name Googlebot or Bingbot. For AI Overviews and Copilot, the fetch evidence still lives in your Cloudflare or server logs. The answer side does not: Promptwatch monitors the real interfaces of ChatGPT, Gemini, Claude, Perplexity, Copilot and others plus Google AI Overviews and AI Mode, so you can see whether those surfaces cited the page. It also connects to Google Search Console, and visitor analytics (a lightweight script or GTM template) shows whether AI-referred sessions converted.
Crawler logs start on Professional at $245 a month with 25M crawler logs. Business is $579 with 100M. Agency plans run from Kick-off at $199 (10M) to Scale at $799 (100M). Essential at $95 doesn't list crawler logs, so pick the tier before you wire up the CDN.
A weekly loop that ends in a change
Errors go first. A 403 or 5xx on a money page is an incident, and no rewrite fixes it. Then blocks: if your robots.txt or a Cloudflare rule keeps out a search bot you want, change it and check the next week's log for the fetch.
Next, the pages that were fetched but not cited. That is where content work earns its keep. Promptwatch's content gap analysis shows which competitor pages win the prompt, and a Content Agent can draft the replacement, hold it in the review inbox, and publish to Webflow or Framer once someone approves it.
Last, the pages that were cited. Check visitor analytics. A citation with no sessions behind it usually means the snippet or the call to action on that URL needs work.
Cloudflare tells you who knocked. A visibility tracker tells you who got quoted. If you want one workspace that connects the knock to the quote to the sale, and then drafts the page that closes the gap, start with Promptwatch on a plan that includes crawler logs.