A new licensing standard, Really Simple Licensing (RSL), has been introduced to enable web publishers to define the terms under which AI system developers can utilize their content.
On Wednesday, prominent organizations including Reddit, Yahoo, Medium, Quora, and People Inc. announced their support for the RSL Standard, an open content licensing protocol. This standard allows publishers to specify how bots should pay to scrape their sites for AI training data, with the goal of collectively influencing AI companies to adopt the framework.
The RSL Standard expands upon the existing robots.txt protocol, which has historically allowed publishers to instruct web crawlers on which parts of their site they can and cannot access. RSL enhances this by enabling websites to include licensing and royalty terms within their robots.txt file. These terms can also be embedded in online books, videos, and training datasets, allowing publishers to seek compensation for their use.
The RSL Collective, a newly formed rights organization, is behind the RSL Standard. It is led by Eckart Walther, a co-creator of the Really Simple Syndication (RSS) standard and former CEO of CardSpring, and Doug Leeds, the former CEO of IAC Publishing and Ask.com. Walther stated, “The goal is to create a new, scalable business model for the web. RSL takes some of those early RSS ideas and creates a new layer for the entire internet where licensing rights and compensation rights are defined.”
The RSL Standard supports various licensing models, including free options. Site owners can request that AI companies pay a subscription fee or a pay-per-crawl fee through the RSL Standard, which would be charged each time an AI bot crawls the website. Additionally, a pay-per-inference fee can be implemented, allowing sites to receive compensation when an AI model references their work to generate a response. Bots crawling sites for other purposes, such as archival or search engine inclusion, can continue as usual.
Several media companies, including Vox Media (parent company of The Verge), News Corp (owner of The Wall Street Journal), and The New York Times, have already established licensing agreements with individual AI companies like OpenAI and Amazon. The RSL Collective aims to streamline this process, enabling any website owner or creator to receive payment for their work without negotiating separate deals.
The success of RSL hinges on the adoption by major industry players, particularly AI companies. AI model builders have faced accusations of disregarding sites’ robots.txt files, and there is currently no straightforward method to track inference fees without their cooperation. The RSL Collective is banking on the collective influence of major web publishers to make adopting the standard more appealing. Leeds stated, “Our job is to go out and get a big group of people to say it’s in your interest, both efficiently, because you can negotiate with everybody at once, and legally, because if you don’t, you’re violating everybody at once.”
Unlike systems like Cloudflare’s “pay per crawl,” the RSL Standard alone cannot block bots from visiting a website. The RSL Collective is collaborating with Fastly, a content delivery network, to control AI bot access to websites based on whether they have agreed to license content. Leeds described Fastly as “the bouncer at the door to the club, and they won’t let people in unless they have the right ID. RSL is issuing the IDs. So we say, ‘Hey, you’ve agreed to license this content,’ and Fastly says, ‘Come on, in your ID checks out.’” Publishers not using Fastly can still ask AI companies to license their content but will be unable to block AI crawlers until more providers develop a solution.
Leeds believes that the RSL Collective can legally enforce licenses, stating that “all participants in the collective rights organization participate in the enforcement of any infringement,” thereby spreading legal costs. He compared this system to existing digital rights organizations, such as the music rights group ASCAP, which collects and distributes licensing fees to its members. While conventional music licensing benefits from strong legal precedent for copyright protection, unauthorized scraping and the use of media for training AI systems remain in a legal gray area. Major AI players are currently facing lawsuits from entities like Reddit, Getty Images, and various online publishers.
Leeds and Walther stated, “There has always been a question of whether bots have agreed to terms that they don’t see. RSL changes that fundamentally, putting crawlers on notice of what the terms are before they access a site.”
Leeds hopes the system will create an intuitive way to navigate licensing works for AI training, stating, “What we’re doing is not reinventing wheels or inventing wheels — we’re just bringing them to a place that they haven’t existed before. The reason they haven’t existed here before is because they haven’t had a standard that we could build on. So that’s why RSL Standard is so important: it gives the infrastructure to then create the things that have worked in every other media industry that hasn’t happened yet.”
The RSL Collective is free for publishers and creators to join. Other major brands like O’Reilly, wikiHow, and Ziff Davis (owner of IGN) have also joined the organization.
