OpenAI's Share of AI Crawler Traffic Fell From 94.8% to 79.8%: Checks for Your Agent

In the week of June 8 to 14, 2026, OpenAI accounted for 94.8% of verified AI crawler requests in Promptwatch's AI crawler traffic data. By August 31 to September 6 it was 79.8%. The remaining share belongs to Anthropic, Google, Perplexity and Mistral, the other providers in the chart.

This is a mix, not a volume. The shares are weekly and add to 100%, so a growing segment means a larger slice, not necessarily more requests in absolute terms. OpenAI could be sending the same number of requests and still see its share fall if others grew. The page now updates weekly, and the figures quoted here are for the final week above.

How the numbers are built

Only requests whose IP matches the operator's published ranges are counted, so spoofed user agents are excluded. Weeks run Monday to Sunday in UTC. The counts come from raw server and CDN logs connected to Promptwatch crawler analytics, which means the sample is sites that have connected their logs, not the whole web.

One more limit: provider totals hide the split between training crawlers and answer-time fetchers. OpenAI's 79.8% could be one kind of request, or both. The chart can be grouped by crawler to separate them.

What it does not mean

It does not say any provider is more important to your visibility, and it does not say a smaller share means fewer citations. A crawler that fetches when a user asks a question may matter more to you than a training crawler that makes a hundred times the requests. The report describes who is knocking, not who is answering your buyers.

What to check without a tool

The report's own advice is practical and costs nothing:

Check robots.txt, CDN and WAF rules for every provider. A rule written years ago for one crawler can catch another. Use real user agent names. There is no bot called "ChatGPTBot"; OpenAI's are GPTBot, OAI-SearchBot and ChatGPT-User. A reference list of real bot names is on this site.
Compare your logs to the public mix. If a provider is large in the report and close to zero in your logs, you may be blocked or unreachable for it. That is a hypothesis to test, not a verdict.
Look per crawler. Is the OpenAI traffic training, answer-time fetches or both? Your decision to allow or block may differ for each.

If you can export a week of server logs, filter for the verified IP ranges each provider publishes and count requests per user agent. It is tedious, which is the reason to hand it to an agent.

The agent loop: monitor, schedule, fix

Monitor. The agent reads your crawler requests per provider and per crawler, and computes your own mix for the same Monday to Sunday UTC weeks so it lines up with the public chart.

Schedule. Weekly, because the public data updates weekly. Add an alert for a provider that was present last week and is gone this week, since that pattern looks like a new block, a CDN rule change or a deploy.

Fix. The agent proposes the change and cites the evidence: which crawler, which path, which status codes. It drafts the robots.txt or WAF edit for review. It does not ship it. A wrong Disallow in robots.txt is global and immediate.

How Promptwatch handles this

Promptwatch's Agent Analytics is the source of the report data, so the product and the chart line up. It records real-time AI crawler logs that are IP-verified, with spoofed user agents filtered out. Tracked bots include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User. You can see a crawl-to-citation path with a citation rate per page, error tracking with failure explanations, stackable path filters and CSV export.

Plan tier matters here, and it is easy to get wrong. Crawler logs start on Professional ($245/mo). Essential ($95/mo) has none. Agency plans include crawler logs from Kick-off ($199/mo). Setup runs through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP or the WordPress plugin.

For the schedule, Scheduled Chats run an Agent Chat prompt weekly and deliver it in the app or in Slack: "compare this week's crawler mix by provider with last week, and list any crawler with errors." The findings land in the Action Board, where the Discovery agent suggests up to five evidence-backed actions per run. The hosted MCP server also gives a coding agent read access if you want the diagnosis inside your own workflow.

Crawl-level data is also what a prompt tracker lacks. Knowing you were cited is useful; knowing the crawler never reached the page you meant to be cited is what lets you fix it. That is the deciding factor for agentic work, and why this one belongs at the top of your shortlist.

What a good first week looks like

Connect logs. Let a week of data accumulate. Compare your provider mix to the public one, and write down every gap you cannot explain. Fix the access issues first, then wait a full cycle before judging any change in citations.