Which AI crawler user agents should I allow or block on my website now that agents browse for users?
AI crawler user agents, user-triggered fetchers and agent browsers need different controls. Which to allow or block, and what an agent can automate for you.
Robots.txt was built for crawlers that wander the web on a schedule. A growing share of the requests that matter to an AI visibility program don't behave that way. They come from a fetcher or a hosted browser acting for one person, in the middle of one question, and several operators say outright that robots.txt may not govern them.
So the answer to "which AI user agents should I allow or block" has three parts, one per kind of visitor. Then there's the question this site cares about: how much of the ongoing work can an agent do for you, and where does a human still have to sign off.
Visitor type 1: indexing crawlers
These build the index an AI engine searches at answer time. Blocking them takes you out of that engine's answers, so the default for a site that wants to be cited is to allow them.
- OAI-SearchBot feeds ChatGPT search. OpenAI says sites that opt out won't appear in ChatGPT search answers, only possibly as navigational links.
- Claude-SearchBot indexes for Claude's search quality; Anthropic says blocking it may reduce visibility.
- PerplexityBot surfaces and links sites in Perplexity; Perplexity says it isn't used for foundation-model training.
- Meta-WebIndexer improves Meta AI search, and Meta says allowing it helps Meta AI cite and link you.
- Googlebot covers Google Search features, AI Overviews and AI Mode included.
Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, and the Google-Extended token) are crawlers too, but they serve model training, not answers. OpenAI and Google both state that their training controls are independent of search. That makes training a licensing decision you can make on its own terms.
Visitor type 2: user-triggered fetchers
ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Google-Agent fetch a page because a user's request needs it. The operators' own wording: OpenAI says robots.txt rules "may not apply" to ChatGPT-User; Perplexity says its user fetcher "generally ignores" robots.txt; Meta says its fetcher "may bypass" it; Google says user-triggered fetchers "generally ignore" it. Anthropic is the exception that doesn't carve out Claude-User, and it warns that blocking Claude-User may reduce visibility for user-directed web search.

Google's "Google user-triggered fetchers" page, Google-Agent section, captured November 21, 2026.
Google describes Google-Agent as the fetcher used by agents hosted on Google infrastructure to navigate the web and perform actions on a user's request. If a buyer asks an agent to compare your pricing page with a competitor's and your WAF throws a challenge at Google-Agent, the agent can only read the competitor's page.
Visitor type 3: hosted browsers acting for a user
OpenAI documents a separate case in its commerce docs. For some tasks ChatGPT runs a browser on a computer in the cloud that reads pages and clicks for the user. Those requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421), carry a Signature-Agent header of "https://chatgpt.com", and publish their keys at a well-known directory on chatgpt.com. OpenAI says Akamai, Cloudflare, HUMAN and Vercel verify the signatures automatically, and that these visits are separate from its crawlers, OAI-SearchBot included. Google says it is experimenting with Web Bot Auth under the https://agent.bot.goog identity. Anthropic's hosted browsers identify as Claude-Agent, which it documents in a separate article.
For this class, the control is signature verification at the edge, not a robots.txt group.
What an agent can take off your plate
Most of the recurring work is checking, and checking is where software helps most. In Promptwatch, the pieces fit like this:
- Agent Analytics classifies each request as it arrives and checks it against the operator's published IP ranges, so a fake GPTBot never inflates the count. Since November 4, search indexers like OAI-SearchBot and Meta-WebIndexer count as AI crawls, and only bots that fetch while answering a user, like ChatGPT-User, count as citation crawls.
- Agent Chat sits on the crawler logs page. You can ask which pages returned errors to OAI-SearchBot last week, or why PerplexityBot stopped showing up. If no log data is arriving, it checks the setup status first.
- The Action Board turns site-health findings into prioritized actions, and its Discovery agent investigates on a weekly schedule.
- MCP and the REST API let your own agent pull crawler data into a scheduled briefing. Scheduled Chats can deliver the same summary to Slack.
Crawler logs start on the Professional plan ($245/mo). Two free tools help before that: the robots.txt generator for AI, and the Agentic Readiness Check, which tests robots.txt against nine AI crawlers and also looks at llms.txt, MCP server cards, A2A agent cards, OAuth discovery and Content Signals.
Two other tools in our directory act on the block itself. Cloudflare AI Crawl Control writes your allow or block choice as a WAF rule, free on any Cloudflare plan. Known Agents offers an automatic robots.txt that adds rules for bot categories as new agents appear. Neither runs prompts or measures citations, which is why we rank them below Promptwatch for an AI visibility program.
What still needs a person
The training decision. Whether GPTBot or ClaudeBot may collect your content is a question about licensing and risk, and it belongs to whoever owns those. No tool should flip it for you.
Licensing standards. RSL 1.0, published December 10, 2025 as a Recommendation, lets you point to a license file from robots.txt with a License: directive and permit or prohibit uses such as ai-train and ai-input. We found no AI company committing to honor it, so it states your terms without enforcing them.

The RSL: Really Simple Licensing homepage, captured November 21, 2026.
Edge rules. Changing a WAF or bot-management rule can lock out real users, so someone who owns the CDN should approve it.
Subdomains. Anthropic asks for the opt-out on every subdomain. An agent can list which subdomains still lack rules; a person decides what each one should say.
If you want to see how Agent Analytics, Agent Chat and the Action Board handle your own logs, book a Promptwatch demo and bring your current robots.txt.