AI Crawler Visibility Tool for Bot Activity on Your Website: 'AI Crawlers' in Cloudflare, Ahrefs, Semrush (2026)
Cloudflare, Ahrefs and Semrush all have an AI crawlers view now. What each one actually records, what it lets you do, and why none of them link a crawl to a citation.
"AI crawlers" is now a menu item in three tools a lot of teams already pay for. Cloudflare has one, Ahrefs has one, Semrush has a few. They sound interchangeable and they aren't. One is a gate, one is a server-side bot report, and one is mostly an audit of what you've accidentally blocked.
We rank tools on this site by whether they help you act, not only whether they show you a number. So the comparison below asks the same thing of each: after it shows you that GPTBot hit a page, what can you do next, and can it tell you whether that hit ever turned into a citation or a visit?
Cloudflare AI Crawl Control: the gate
AI Crawl Control, formerly AI Audit, is on every Cloudflare plan. Its Overview, Crawlers and Metrics tabs show requests by crawler and by operator (OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta), allowed versus unsuccessful requests, robots.txt violations, status codes, paths and hosts.
Of the three tools here, this is the only one that changes what happens at the door. You can allow or block each crawler from the same screen, and pay per crawl is in private beta. That counts as acting.
The limits depend on your plan. On Free, detection works off the user-agent string only and the Metrics tab covers the past 24 hours. Referral counts need a paid plan. Enterprise with Bot Management adds detection IDs and configurable timeframes. Everything is also available through the GraphQL Analytics API.
Ahrefs Bot Analytics: the server-side report
Ahrefs Bot Analytics is free while it's in beta. It sorts bots into 12 categories (AI assistants, search engines, SEO tools and so on), gives each bot its own view (Googlebot, GPTBot, ClaudeBot), and shows pages, status codes and site structure, with an "AI bots" toggle to cut the noise.
Data comes from your edge. Ahrefs supports Cloudflare Logpush on Enterprise, a Cloudflare Worker on any plan including Free, a Vercel log drain, Fastly, and an Amazon data stream option. That's a generous set of entry points for a free product.
Two things to keep straight. Bot Analytics is separate from Ahrefs Web Analytics, and it's separate from Ahrefs Brand Radar, which is the product that looks at AI answers. The bot report tells you GPTBot read /pricing. It doesn't know whether ChatGPT then cited /pricing.
Semrush: audit, log upload, and an Enterprise layer
Semrush spreads the job across several products, and which one you get depends on your contract.
Site Audit now has an AI Search Health score and a Blocked from AI Search widget that shows which of 8 major AI crawlers your robots.txt blocks. There's an "AI Search" issues filter, and you can set the audit's user agent to OpenAI-Search. This is a simulation of access, not a record of real visits. It's good at catching the robots.txt line a developer added during a migration and forgot.
The Log File Analyzer is the part that looks at real traffic. Upload or connect server logs and filter for GPTBot, ChatGPT-User, OAI-SearchBot or ClaudeBot.
Semrush Enterprise goes further: Bot Analytics in Site Intelligence covers 30 bots, there's an "Agent Analytics" view inside AI Optimization (AIO), and Crawler Profiles simulate ChatGPT (ChatGPT-User, OAI-SearchBot, GPTBot) and Googlebot crawls. That layer is Enterprise-only with no public price.
A naming note, since buyers mix these up. Promptwatch's crawler product is also called Agent Analytics, and per its founders it shipped first in the category. Same name, different products. The prompt-side Semrush product is the Semrush AI Toolkit, sold separately as a $99 a month per-domain add-on.
Where all three stop
Each tool answers "did the bot arrive, and what status did it get?" Cloudflare also lets you decide whether it may arrive at all. None of them answers the next two questions:
- After ClaudeBot fetched this page, did Claude cite it for the prompt we care about?
- When a visitor came from that answer, did they sign up or buy?
Without those, a crawler report becomes a vanity metric fast. A month of rising GPTBot requests looks like progress until you learn they were training fetches and the prompt you sell on still cites a competitor.
Promptwatch Agent Analytics: the crawl attached to the prompt
Promptwatch runs crawler logs inside the same workspace as prompt tracking, citation analytics and visitor analytics. That's the reason it tops our directory for this use case.
Agent Analytics logs ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther and Meta's AI crawler in real time. It draws the crawl-to-citation path, so you can see which fetches came before a citation, and it tracks errors on their own so a timeout gets fixed as a timeout. Logs arrive from Cloudflare (Logpush on Enterprise, or an auto-deployed Worker on other plans with the record orange-cloud proxied), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or a custom HTTP source.
Then it acts. Citation analytics and content gap analysis show which competitor page wins the prompt. A Content Agent drafts the page you're missing, parks it in a review inbox, and publishes to Webflow or Framer once a human approves. The next week's log shows whether PerplexityBot fetched the new URL. Prompt tracking shows whether Perplexity cited it. Visitor analytics shows whether anyone came through.
Crawler logs start on Professional at $245 a month (25M crawler logs, 150 prompts, 2 projects). Business at $579 has 100M. Agencies get them from Kick-off at $199 (10M) up to Scale at $799 (100M). Essential at $95 doesn't list crawler logs.
Picking by job
| Your situation | Start with |
|---|---|
| You need to block or allow specific AI bots today | Cloudflare AI Crawl Control |
| You want a free, per-bot server report and already use Ahrefs | Ahrefs Bot Analytics |
| You suspect robots.txt is blocking AI search bots | Semrush Site Audit |
| You need to know if a crawl led to a citation and a conversion, then fix the gap | Promptwatch |
These aren't exclusive. Cloudflare's controls and Promptwatch's logs sit on the same edge without conflict, and plenty of teams keep a Semrush audit running for the robots.txt check. But if you're buying one tool to answer "is our AI crawler activity paying off," the gate and the bot report won't get you there. Evaluate Promptwatch first, on a plan with crawler logs, and judge it on whether the first gap it finds turns into a published page.