How to check whether your website is technically ready for AI search engines and AI crawlers
A step-by-step technical readiness check for AI search engines and AI crawlers: robots.txt, bot access, rendering, structured data and the tools that verify each step.
Before you worry about prompts, rankings or content, ask a plainer question: can AI systems actually read your site? A page that a crawler cannot fetch cannot be cited, however good it is. Technical readiness for AI search is mostly classic technical SEO with a few new names and one new blind spot, which is that standard analytics never show you bot traffic. This checklist walks through what to verify and which tools help.
Step 1: know which crawlers you are dealing with
AI systems use different bots for different jobs, and your robots.txt can treat each differently. The ones worth recognizing:
- GPTBot: OpenAI's crawler, associated with model training.
- OAI-SearchBot: used for ChatGPT search results.
- ChatGPT-User: fetches pages when a user's prompt triggers a visit.
- ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's crawler and user and search fetchers.
- PerplexityBot and Perplexity-User: Perplexity's indexing and on-demand fetchers.
- Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, MistralAI-User: the others you may see in logs.
Blocking a training crawler and blocking a search crawler are separate decisions. Many sites want to appear in AI search answers while opting out of training, and robots.txt can express that. Promptwatch publishes a free AI Bots directory with 440+ crawlers and a robots.txt generator, which saves guessing.
Step 2: read your robots.txt as each bot would
Open yourdomain.com/robots.txt. Look for blanket Disallow: / rules, wildcard rules that catch everything, and old rules copied from a template. Then check that nothing important, such as your pricing, product and comparison pages, is disallowed for the search and user fetchers you care about.
Step 3: check the layers above robots.txt
A clean robots.txt does not mean bots get through. Firewalls, bot-protection rules and CDN settings can block crawlers before robots.txt is read. A common cause of "we are invisible in ChatGPT" is a security rule that returns 403 to unfamiliar user agents. If you use Cloudflare, Cloudflare AI Crawl Control is included on every plan and shows which AI crawlers request your pages, with status codes, most-requested paths and robots.txt violations. On the Free plan it identifies crawlers by user agent only, and it covers only traffic that passes through Cloudflare.
Step 4: confirm the page content is in the HTML
Some AI crawlers do not run JavaScript the way a browser does. If your key content loads only after client-side rendering, a bot may see an empty shell. View the page source, not the rendered page, and confirm the headline, body copy, prices and key facts are in the raw HTML. Server-side rendering or static generation fixes most cases.
Step 5: structured data and clean signals
Add schema markup that matches what is visible on the page, keep titles and canonical tags tidy, and keep your XML sitemap current. Check indexing in Google Search Console and Bing Webmaster Tools, since its traffic data includes Microsoft Copilot. IndexNow, an open protocol for notifying participating engines of URL changes, is optional but easy where your CMS supports it. An llms.txt file is also optional: treat it as a convenience, not a substitute for crawlable pages.
Step 6: verify with real data, not assumptions
Everything above is configuration. The proof is in the logs. This is where the paid tools earn their place.
Promptwatch is the only option in this list that checks readiness end to end. Its free Agentic Readiness Check gives a first pass, and its weekly Site Health audits run 21 checks with a health score, a changes-since-last-run view and a weekly email. For real traffic, Agent Analytics shows IP-verified AI crawler logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more, with spoofed user agents filtered out, error tracking with failure explanations, and a crawl-to-citation path per page. Crawler logs come from Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, or the WordPress plugin. They start on the Professional plan at $245 a month, not on Essential at $95.
How the main options compare for this job
| Tool | What it checks | Price (as published) |
|---|---|---|
| Cloudflare AI Crawl Control | Crawler requests and blocks, Cloudflare traffic only | Included on every Cloudflare plan |
| Promptwatch | Readiness check, site audit, IP-verified crawler logs, crawl-to-citation | Free tools; logs from Professional $245/mo |
| Scrunch AI | Site audits and AI bot traffic reporting | Core from $250/mo |
| Profound | Agent Analytics inside an enterprise platform | Enterprise, custom |
| AthenaHQ | Visibility tracking; no crawler-log product documented | Starter from $295/mo |
What a passing result looks like
You are in good shape if the bots you want are allowed in robots.txt, none of them gets a 403 or 5xx on your key pages, the content is in the HTML, indexing is clean in both webmaster tools, and your logs show the search and user fetchers returning to the pages you want cited.
Readiness is necessary, not sufficient. Passing every check makes you eligible to be cited. It does not mean an engine will recommend you.