A Reference List of Real AI Bot Names for Your robots.txt in 2026
Which AI crawler user agents actually exist, what each is for, and how to write robots.txt rules that do what you intend.
Most robots.txt mistakes about AI are naming mistakes. A rule written for a bot that does not exist protects nothing and blocks nothing, and the person who wrote it believes the problem is handled. The commonest example is "ChatGPTBot," which is not a real user agent. If you see it in a guide or a template, the guide was not checked.
This post lists the user agents that Promptwatch's Agent Analytics tracks, explains what each family is used for, and then walks through the decisions you actually need to make.
The names that are real
These are the bots tracked in Promptwatch's crawler logs:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Agent, GoogleOther
- Meta: Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher
- Mistral: MistralAI-User
Promptwatch also runs a free AI Bots directory that lists more than 440 crawlers, which is a better reference than any list in a blog post, including this one. Bookmark it and check names against it before writing a rule.
Three jobs, three kinds of bot
The names above fall into three rough jobs, and the pattern in the names helps. Treat this as a reading aid, not as a formal specification, and check each vendor's own documentation for exact behavior.
Training or broad collection. Bots like GPTBot and ClaudeBot are generally associated with gathering content for models. Blocking them is about model training preferences.
Search indexing. Bots like OAI-SearchBot and Claude-SearchBot are associated with building a search index that powers cited answers. Blocking these can reduce the chance of being cited in that engine's search results.
User-triggered fetching. Bots like ChatGPT-User, Claude-User, Perplexity-User, and MistralAI-User are named for fetches made on behalf of a person's request. These are the ones that tend to show up when somebody asks a question about your page.
The practical point: blocking all of them with one broad rule is a decision to leave AI search, not just training. If you want to be cited, think carefully before you block the search and user-fetch bots.
Writing rules that work
Each User-agent line in robots.txt matches one bot name. A few rules of thumb:
- Use the exact token the vendor documents.
- Put specific groups before a catch-all group so you do not rely on ordering quirks.
- Test the file. Fetch it as the bot would and confirm the response.
- Remember that robots.txt is a request, not a wall. A badly behaved crawler may ignore it. That is why IP verification matters when you read logs.
A sample intent, in words rather than syntax: allow the search and user-fetch bots for the whole site, decide separately about training bots, and keep private paths blocked for everyone.
The spoofing problem
User agent strings are text, and anyone can send any text. A scraper can claim to be GPTBot. If you judge AI crawl activity by user agent alone, your numbers include impostors, and you may waste time on a bot that is not who it says.
Agent Analytics in Promptwatch is real-time and IP-verified, so spoofed user agents are filtered out. That matters for decisions: you want to know that the GPTBot requests you see came from the real thing before you conclude that your pages are being fetched.
The check that matters more than the rule
Writing a rule is half the job. The other half is confirming what actually happens. After any change to robots.txt, check crawler logs over the next week. Look for three things: do the bots you allowed show up, do they fetch the pages you care about, and do the requests succeed?
Error tracking in Agent Analytics reports failures with explanations and supports stackable path filters, so you can narrow to one section of the site. It also follows the crawl-to-citation path, with a citation rate per page, so you can tell whether being fetched leads to being cited. Logs can be exported as CSV.
Crawler logs come from your CDN or host. Promptwatch supports Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, and the WordPress plugin. They start on the Professional plan. Essential does not include them.
Common mistakes
- Blocking a bot name that does not exist.
- Blocking the search bot while expecting citations from that engine.
- Forgetting that a firewall or bot-protection layer can reject bots even when robots.txt allows them.
- Assuming a rule worked without checking logs.
- Never rereading the file after a site migration.
An agentic way to maintain it
A rule file goes stale. New bots appear and old ones change. A sensible routine is monthly: compare the user agents in your logs with your rules, and flag any bot that appears in logs but has no rule, or any rule that names a bot you never see.
You can have an agent do the comparison. Promptwatch exposes a hosted MCP server and a REST API, so a coding agent can read crawler data, and the Action Board can hold the resulting change as a task. A human should approve any edit to robots.txt, since one wrong line can block your whole site.
If you need the file itself, Promptwatch offers free llms.txt and robots.txt generators.
The recommendation
Use real names, verify them against a directory, and confirm results in logs. Promptwatch gives you the names, the IP-verified logs, and the crawl-to-citation path in one place, which turns robots.txt from a guess into a measurable decision. Start by listing which of the bots above appear in your logs today, then write your rules around what you find.