The Trust Problem in AI: Why Companies Can’t Grade Their Own Homework

The recent spate of high-profile incidents involving OpenAI, Anthropic, and Meta has exposed a fundamental flaw in the way frontier AI research is conducted. These companies are responsible for developing powerful AI models while also evaluating their safety – a task that essentially requires them to grade their own homework.

This lack of transparency and accountability is not surprising given the enormous stakes involved. As these models become increasingly sophisticated, they pose catastrophic risks to society, including job displacement, cyber attacks, and potential existential threats. Despite these issues, there are no independent institutions in place to verify the safety protocols of these companies or ensure that their testing methods are sufficient.

The system currently in place is unsustainable as these models become more capable and potentially disastrous consequences become increasingly likely. Companies racing to build the world’s most powerful AI models get to decide which failures matter and what risks they can safely ignore.

Frontier AI safety expertise resides almost exclusively within companies like OpenAI, Anthropic, and Meta. While these organizations deserve credit for disclosing recent incidents, a system that depends on voluntary transparency is inherently fragile and prone to abuse.

The FRONTIER Act offers a potential solution by establishing licensed Independent Verification Organizations (IVOs). These IVOs would be technical experts outside the AI labs who evaluate whether companies’ safety frameworks keep catastrophic risks within acceptable bounds. They would create a market for independent oversight and verification, attracting engineers, cybersecurity researchers, evaluators, auditors, and eventually insurers.

Independent verification would provide a trusted signal that frontier AI systems have been evaluated by someone other than the company seeking to deploy them. It would also make it possible for insurance markets to emerge around frontier AI risk – currently difficult because insurers lack a trusted third-party basis for evaluating catastrophic risks.

The missing piece is not technical talent; organizations like METR, Apollo Research, SecureBio, and leading cybersecurity firms already possess relevant expertise. What’s needed is a system that requires and rewards independent evaluation.

In every other high-stakes industry, independent institutions exist to ensure public safety doesn’t depend on voluntary transparency. It’s time AI was held to the same standard. The next time a frontier AI model behaves in an unexpected or dangerous way, the public shouldn’t have to hope the company involved decides to disclose it.

The FRONTIER Act offers a crucial step towards creating independent oversight and verification – a necessary safeguard against the catastrophic risks posed by these powerful models. If lawmakers create such a system, it could mark a turning point in the development of AI research – one where companies are held accountable for their actions and the public can trust that catastrophic risks are being taken seriously.

If we fail to act, we risk repeating the mistakes of the past, where powerful technologies were developed with little regard for their consequences. The stakes are too high to ignore this warning sign any longer. It’s time to create a system that puts public safety above all else – a system that recognizes AI labs shouldn’t be allowed to grade their own homework.