A new licensing initiative seeks to empower web publishers by allowing them to define the conditions under which AI developers can utilize their content. Major platforms such as Reddit, Yahoo, Medium, Quora, and People Inc. have endorsed Really Simple Licensing (RSL), an open standard that gives publishers the ability to specify compensation terms for the scraping of their sites for AI training purposes. This cooperative effort aims to enhance their bargaining power with AI firms.
The RSL Standard extends the functionality of the existing robots.txt protocol, which has traditionally enabled publishers to communicate with web crawlers about access permissions to their sites. With RSL, website operators can now enrich their robots.txt files with detailed licensing and royalty clauses regarding how AI systems can utilize their content.
The initiative is spearheaded by the newly established RSL Collective, led by Eckart Walther, a co-inventor of the RSS standard, and Doug Leeds, former CEO of IAC Publishing and Ask.com. Walther shared in an interview with Technology News, “Our objective is to forge a new, scalable business model for the web that lays down clear licensing and compensation rights across the internet.”
Under the RSL Standard, various licensing models are available, including options for free access. Publishers may charge AI companies through subscription models or impose fees for individual crawls and inferences made by AI bots. Additionally, the standard allows for a pay-per-inference structure, enabling publishers to earn compensation when their content is referenced by AI-generated outputs. Crawlers intended for non-commercial purposes, such as those used for archiving or search engines, will continue to operate without modification.
“We are not reinventing wheels or creating new ones.”
Moreover, several media entities, including Vox Media, News Corp, and The New York Times, have already formed licensing agreements directly with AI developers like OpenAI and Amazon. In contrast, the RSL Collective strives to streamline this process, allowing any website owner to easily establish payment mechanisms for their content without the need for lengthy negotiations.
The effectiveness of the RSL Standard hinges on the support from major AI companies. Past instances have shown that AI developers often overlook robots.txt directives, complicating the tracking of fees without their involvement. Leeds asserts that uniting significant web publishers will incentivize AI firms to adopt the standard for mutual benefit. He stated, “Our mission is to coordinate a collective of stakeholders to demonstrate that compliance is both convenient and legally sound.”
While the RSL Standard does not independently block bots from accessing websites, unlike Cloudflare’s pay-per-crawl model, the RSL Collective is collaborating with Fastly to manage AI bot access contingent on content licensing agreements. Leeds likens Fastly to a “bouncer,” only permitting entry to bots that can present valid licensing verification from RSL.
Leeds expresses confidence that the RSL Collective can enforce licensing agreements as all members contribute to addressing any legal infringements. He draws parallels to established digital rights organizations, such as ASCAP, which facilitates licensing fee distribution to its members. Despite the existing legal ambiguities surrounding the scraping of content for AI training, the RSL Collective aims to create a more defined structure for publishers.
The RSL Collective argues that it fundamentally alters the agreement dynamics for AI bots by explicitly notifying them of the terms before they engage with the content. “The RSL framework sets clear parameters for bots, ensuring they understand the rules of engagement prior to site access,” Leeds and Walther emphasized in a joint statement.
Despite current challenges, Leeds is optimistic about establishing a straightforward licensing framework for AI training content. “What we are creating is neither new nor revolutionary; it merely adapts existing practices to a new landscape,” he stated. “This initiative offers the infrastructure necessary for models proven successful in other media sectors to thrive in the digital realm.”
Joining the RSL Collective is free for publishers and creators, with major entities like O’Reilly, wikiHow, and Ziff Davis, the owner of IGN, also participating.
