What is RSL (Really Simple Licensing), and should publishers use it to control how AI crawlers and agents use content?
RSL gives AI agents machine-readable licensing terms. What the standard automates, where robots.txt stops reaching agents, and which crawler checks can run on their own.
Most writing about RSL treats it as a publishing rights story. It is also an automation story. Really Simple Licensing exists because the readers of the web are increasingly machines that act on their own: crawlers that build indexes, fetchers that load a page because a user asked, and browsers that click through a site for someone. RSL tries to give those machines terms they can read and, in its fuller form, a way to obtain a license without a human negotiating each deal.
What the standard does
RSL is an open standard: a way for a site to spell out what AI systems and automated agents are allowed to do with its content, and under what license. The current version, RSL 1.0, carries "Recommendation" status (published December 10, 2025, previous version 0.9). A site points to an XML license file from robots.txt with a License: directive, or from a Link response header, a link tag in the HTML, embedded RSL, or an RSS feed. The license can allow or forbid specific uses, ai-train and ai-input being two of them, and name a payment model.
The automated part is the Open License Protocol. A license server issues tokens, so an AI company's systems can request and receive a license programmatically. A Crawler Authorization Protocol governs access, and the Encrypted Media Standard covers content that is not public. When a license is missing, a server may respond with HTTP 401 or 402.
None of the AI vendors' pages we read say their crawlers honor RSL. Until one does, the machine-to-machine licensing loop only works with AI companies that opt in.
Why robots.txt is not enough for agents
Here is the gap RSL's enforcement layer aims at. Several vendors say their user-triggered agents do not follow robots.txt the way crawlers do:
- OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a user initiated the visit.
- Perplexity says Perplexity-User "generally ignores robots.txt rules".
- Google says user-triggered fetchers, Google-Agent among them, generally ignore robots.txt.
- Meta says Meta-ExternalFetcher "may bypass robots.txt rules".

Meta for Developers, "Meta Web Crawlers", captured November 21, 2026.
Anthropic's help page is the exception in this group: it says its bots honor robots.txt and does not carve out Claude-User. Separately, Anthropic recognizes a machine-readable "no AI training" preference signal based on the IETF aipref draft as of October 30, 2026, and Google says it intends to support the AIPREF training signal in early 2027. Those are preference signals for training, not licenses.
For agents that ignore robots.txt, a declared license in the same file is just as easy to ignore. Only a server-side response, such as a 402 to an unlicensed request, actually blocks the request. That requires knowing which requests are which. OpenAI signs its cloud browser's requests with Web Bot Auth, and Google is experimenting with the same protocol for its agent identity, which helps.
What you can automate
The monitoring around a licensing decision is where agents earn their keep.
Promptwatch Agent Analytics records AI crawler visits in real time, checked against the IP ranges each operator publishes, and since November 4, 2026 it classifies search indexers as AI crawls and live fetches like ChatGPT-User as citation crawls. A Host column flags logs from hostnames that are not your project website. Crawler logs start on Professional at $245 a month.
On top of that data:
- A Scheduled Chat in Agent Chat can post a weekly crawler briefing to a Slack channel, so a shift in which bots visit shows up without anyone opening a dashboard.
- The Action Board includes site health among its action types, and its Discovery agent investigates on a weekly schedule and proposes up to five evidence-backed actions per run.
- The MCP server lets your own agent in Claude, Cursor or ChatGPT pull crawler logs and citations into whatever workflow handles your licensing data.

Promptwatch bot directory, GPTBot page, captured November 21, 2026.
Promptwatch's free robots.txt generator for AI and the Agentic Readiness Check (which tests robots.txt against nine AI crawlers and looks at llms.txt, MCP server cards, A2A agent cards, OAuth discovery and Content Signals) cover the setup side.
What needs a person
Licensing terms and prices are commercial choices. So is the decision to let OAI-SearchBot and Claude-SearchBot index you for answers while signaling no training to GPTBot and ClaudeBot. Whether RSL terms are enforceable in your jurisdiction is a legal question no tool answers, and none of our sources settle it.
Tools that act on crawler data, ranked
Promptwatch leads because it pairs verified crawler logs with citations and AI traffic, then lets agents act on them. AthenaHQ automates optimization tasks but publishes no crawler-log product. Scrunch AI lists bot traffic reporting and, on Enterprise, AXP for crawler-specific page versions. Goodie AI lists crawler analytics and runs content agents, from $295 a month. Profound sells broader marketing agents on custom contracts.
To see which agents your terms would address, try Promptwatch free for seven days on a monthly Professional plan and set a weekly crawler briefing in Slack on day one.