Agentic Web Optimization: A Brand Guide
A 30-day plan for making your site work for AI agents, split into what to automate and what to keep manual: crawler access by purpose, product feeds, UCP and ACP triage, and measurement.
Most advice about optimizing for AI agents is a list of things to check. A list doesn't tell you who does the checking, or which parts a machine can do for you. This plan does. It runs 30 days, and every task gets one of three labels based on how much work a tool can take off your plate:
- Monitor: a tool collects the data, a person reads it.
- Recommend: a tool proposes the change, a person decides.
- Execute: a tool makes the change, usually after a person approves it.
Some tasks should stay fully manual no matter how good the tooling gets. Deciding whether AI companies can train on your content is one. Anything touching payments is another.
Days 1 to 7: find out which agents actually visit
Keep manual: the access policy. Write one page that sorts AI agents by purpose, because each vendor splits them that way and blocking one is not like blocking another.
Search crawlers put you in answers. OpenAI's OAI-SearchBot surfaces sites in ChatGPT search. Anthropic's Claude-SearchBot indexes for Claude's search quality. PerplexityBot surfaces and links sites in Perplexity search, and Perplexity says it isn't used for training. Blocking any of them costs you a surface.
Training crawlers are GPTBot (OpenAI) and ClaudeBot (Anthropic). Whether to allow them is a licensing decision for legal, not a ticket for SEO. Google-Extended sits nearby but is different: Google says it has no user agent of its own and is a robots.txt control token covering Gemini training and grounding, with no effect on Google Search inclusion.
User-triggered agents fetch a page because a person asked. These are ChatGPT-User, Claude-User and Perplexity-User. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores them. Claude-User respects robots.txt, as do all three Anthropic bots, according to Anthropic, which also notes each bot must be blocked separately per subdomain.
Automate: log collection. Robots.txt shows intent. Logs show what happened. Connect Promptwatch Agent Analytics to your CDN (Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or custom HTTP) and let it record AI crawler hits, the pages they request, the errors they hit and the crawl-to-citation path. Label: monitor. By day 7 you should know which agents show up and where they fail.
Days 8 to 14: fix access and facts
Recommend, then ship by hand: robots and firewall changes. An agent can read a week of logs and draft the robots.txt change; we walked through that loop in letting an agent read crawler logs and propose robots.txt fixes. A person reviews and deploys it. Two things to know before you do. User agent strings can be faked, so verify against published IP lists: OpenAI publishes them for OAI-SearchBot, GPTBot and ChatGPT-User, and Perplexity for both of its agents, which it recommends matching together with the user agent in firewall rules. And allow a day before you judge the result. OpenAI says search robots changes take about 24 hours, Perplexity up to 24.
Keep manual: the facts pass. Pick your top commercial pages and check that products, prices, plan names and policies say the same thing on the page, in the feed and on partner listings. Facts belong in text, not images. One name per product. Return policy and support contact on crawlable pages. A tool can flag mismatches, but the person who owns pricing has to decide which version is right.
Automate: citation tracking. Promptwatch citation analytics shows which of your pages get cited, and which Reddit threads, YouTube videos and third-party domains get cited instead. Label: monitor. This tells you which fixed pages are worth watching.
Days 15 to 21: feeds and protocol triage
Automate where your platform already does it: feeds. On 24 March 2026, OpenAI extended ACP (the Agentic Commerce Protocol, built with Stripe) to product discovery. Merchants can share feeds and promotions, including through Salesforce and Stripe. Shopify merchants are covered through Shopify Catalog with no extra work. The ACP product spec asks for a stable id per product and at least one variant with its own stable id and title, with price, media, GTIN and URLs optional but helpful. For Google, a Merchant Center account in good standing is the base layer, and Google recommends a supplemental feed for the checkout attributes.
Keep manual: protocol decisions. Four names will land on your desk.
UCP is Google's open standard for discovery, buying and post-purchase support. Its checkout puts a Buy button on eligible listings in AI Mode and Gemini while you remain merchant of record. It's "available for select merchants at this time" through early access, for products eligible in the US, Canada and Australia. Subscriptions, pre-orders, used goods, services and age-restricted items are among the exclusions. The native_commerce(checkout_eligibility) attribute defaults to FALSE.
ACP, on the checkout side, has shifted. OpenAI says it is moving away from a standalone Instant Checkout, and purchases now complete on merchant-owned sites or apps.
AP2 proves a user authorized an agent to buy. Version 0.2 was donated to the FIDO Alliance. Watch it; don't build on it yet unless a payments partner asks.
MCP connects AI apps to tools and data. It isn't a checkout protocol. It matters to you as the way an agent could reach your own tools, and the way you'll pull your visibility data in the last week.
Automate: shopping visibility. If you sell products, add them to Promptwatch's ChatGPT Shopping tracking. Over MCP the relevant tools are addShoppingTrackedProducts and getShoppingProductPositionAnalytics. Label: monitor.
Days 22 to 30: close the loop
Execute with approval: gap content. Promptwatch's content gap analysis finds prompts where competitors are cited and you aren't. Content Agents plan and write the page, drop it in a review inbox, and publish to Webflow, Framer or WordPress once a person accepts it. One quirk: publishing to Framer redeploys the whole Framer site, so batch those approvals. Label: execute, with a human on the publish button.
Automate: traffic and conversions. Install Promptwatch visitor analytics with its script or the GTM template so AI-referred visits and conversions are counted. ChatGPT adds utm_source=chatgpt.com to referral URLs, but that's one assistant and it doesn't tie to conversions on its own.
Automate: reporting. Pick one route. Connect the Promptwatch MCP server (https://server.promptwatch.com/mcp) to Claude as a custom connector and run a saved prompt weekly that calls getCrawlerTrend, getTopCrawlerPages, getCitationTopPages and getVisitorReferrerStats; a read-only key hides the write tools. Or use the Promptwatch Slack agent, the Looker Studio connector (Professional, Business and agency plans), or REST API v2 for scripted pulls.
The split at a glance
| Task | Automate | Keep manual | Label |
|---|---|---|---|
| Training crawler policy | Yes | Manual | |
| Crawler log collection | Yes | Monitor | |
| Robots.txt and firewall edits | Draft | Deploy | Recommend |
| Fact consistency | Flag mismatches | Decide | Recommend |
| Citation tracking | Yes | Monitor | |
| Product feeds | Via platform | Execute | |
| UCP, ACP, AP2 decisions | Yes | Manual | |
| ChatGPT Shopping tracking | Yes | Monitor | |
| Gap content | Draft and publish | Approve | Execute |
| Weekly reporting | Yes | Read it | Monitor |
Promptwatch covers every automated row in that table except the feeds, from crawler logs through publishing, which is why it sits at the top of our rankings. Crawler log volume is listed from Professional at $245/mo (25M crawler logs, 5 AEO articles). ChatGPT Shopping tracking and the automated Content Agent are on Business at $579/mo. Essential at $95/mo has 50 prompts, 200K visitor events and a 7-day trial. Annual billing is 12 months for the price of 10. Run the 30 days on promptwatch.com.