In September of last year, California Senate Bill 1047 garnered significant attention as it was sent to Governor Gavin Newsom, who ultimately chose to veto it. The legislation aimed to mandate that developers of large AI models, especially those with training costs exceeding $100 million, perform tests to identify potential dangers.
The governor’s veto was met with disappointment from AI industry whistleblowers, while established tech companies expressed relief. However, the discussion surrounding AI regulations did not conclude there. Newsom expressed concerns regarding the legislation’s strictness and commissioned a panel of prominent AI researchers to develop an alternative strategy. This new proposal would aim to balance the advancement of generative AI technology with appropriate regulatory frameworks to mitigate associated risks.
On Tuesday, a report outlining this new approach was released.
Titled the “California Report on Frontier Policy,” the 52-page document highlights that advancements in AI capabilities, including the reasoning skills of models, have accelerated since the governor’s veto of SB 1047. By employing historical examples, empirical studies, and simulations, the authors proposed a new framework emphasizing increased transparency and independent evaluation of AI models. This report arrives amid discussions regarding a potential 10-year moratorium on AI regulations at the state level, supported by a Republican-majority Congress and influential companies like OpenAI.
The report was co-chaired by Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence; Mariano-Florentino Cuéllar, President of the Carnegie Endowment for International Peace; and Jennifer Tour Chayes, Dean of the UC Berkeley College of Computing, Data Science, and Society. It identified that groundbreaking AI developments in California could have significant effects on multiple sectors, including agriculture, biotechnology, clean technology, education, finance, medicine, and transportation. The authors emphasized the necessity of nurturing innovation while ensuring that regulatory measures do not impose excessive burdens on organizations.
“Without proper safeguards… powerful AI could induce severe and, in some cases, potentially irreversible harms.”
The report stresses the urgency of addressing risks associated with AI: “Without proper safeguards… powerful AI could induce severe and, in some cases, potentially irreversible harms.”
While a draft version of this report was made available for public feedback back in March, the final report indicates a growing concern about AI models’ contributions to risks related to chemical, biological, radiological, and nuclear (CBRN) weapons. The authors noted that leading companies had reported increases in their models’ capabilities in these areas.
Revisions have been made in the final report to reflect the need for California’s AI policies to adapt to evolving geopolitical conditions. The authors have provided further context on the risks posed by large AI models and have shifted their focus on regulation criteria, stating that scrutinizing companies solely based on their computational requirements for training is insufficient.
The authors pointed out that AI training requirements are continually changing, and a compute-centric definition overlooks how these models are utilized in practical situations. They believe this definition can serve as an initial filter for identifying entities that may require closer examination, but that evaluative measures encompassing initial risks and aftermath impacts are just as vital.
The need for transparency in the AI industry remains critical, as the report indicates. With little consensus on best practices and significant opacity regarding aspects like data acquisition, safety protocols, pre-release testing, and the potential downstream consequences, the authors stress the importance of clearer disclosure practices.
The report advocates for the establishment of whistleblower protections, third-party evaluations with safe harbor provisions for researchers, and enhanced public information sharing to foster greater transparency beyond the current disclosures by major AI companies.
Scott Singer, one of the lead authors, remarked to Technology News that discussions around AI policy have significantly evolved at the federal level since the draft was released. He believes that California could play a vital role in aligning state regulations with common-sense approaches that resonate with a broad audience, contrasting with the disparate laws that may arise from AI moratorium advocates.
Earlier this month, Anthropic CEO Dario Amodei called for a federal standard on transparency in an opinion piece, advocating for leading AI firms to make public their strategies for identifying and mitigating national security threats and other catastrophic risks.
“Developers alone are simply inadequate at fully understanding the technology and, especially, its risks and harms.”
The authors emphasize that relying solely on developers to comprehend the risks associated with this rapidly evolving technology is insufficient. Hence, third-party risk assessments are deemed essential within the report.
The report suggests that such evaluations would motivate companies like OpenAI, Anthropic, Google, and Microsoft to enhance safety measures for their models while aiding in a clearer understanding of the associated risks. Currently, evaluations are mostly conducted internally or by contractors hired by the companies themselves, which the authors argue is not sufficient.
The authors point out that there are “thousands of individuals” eager to engage in risk assessments, a number far exceeding internal or contracted teams. Moreover, these third-party evaluators bring a diverse range of perspectives that often reflect communities adversely affected by AI technologies.
However, effective risk assessments require ample access to proprietary models, which companies are often reluctant to provide.
Challenges in access have also been noted for second-party evaluations. Metr, a collaborator with OpenAI, reported in a blog post that it faced limited time and access to data when testing OpenAI’s o3 model, hampering its ability to conduct comprehensive assessments. Subsequently, OpenAI expressed intentions to expand data sharing with partners like Metr.
The report further indicates that mere API access or model weight disclosures may not be sufficient for meaningful evaluations, as companies might rely on restrictive terms of service to deter independent researchers from identifying safety vulnerabilities.
In a March open letter, over 350 AI industry researchers called for a “safe harbor” for independent AI safety assessments, similar to existing protections for third-party cybersecurity evaluators. The recent report reiterates this call for significant changes and introduces options for reporting incidents involving AI systems.
The authors concluded by warning that even the most well-designed safety policies cannot completely eliminate substantial negative outcomes. They stressed that as foundation models gain wider adoption, it becomes increasingly crucial to understand the real-world harms they may cause.
