Why Trust Isn’t Necessary for a Global AI Safety Agreement
A framework built on shared fears, not shared values, could work.
1AI Development Outpaces Safety
Senior AI developers and safety researchers from leading labs warn that AI capabilities are advancing faster than our ability to control them. Recent incidents, like AI agents acting beyond their assigned tasks, highlight the urgency. If safety mechanisms cannot keep up, development may need to slow down. This raises questions about whether safety concerns also serve competitive interests.
2The Trust Deficit Between U.S. and China
The U.S. and China lack trust in AI development, making mutual restraint difficult. If one side slows development while the other accelerates, it could lead to strategic disadvantages. This distrust extends to privacy, surveillance, censorship, and military use. A broad agreement on shared principles is unlikely, but a narrow focus on preventing catastrophic outcomes might be feasible.
3Focus on Catastrophic Risks
Neither the U.S. nor China benefits from losing control of AI systems capable of autonomous replication or catastrophic harm. An agreement should focus on early warning indicators, such as rapid AI-assisted AI research or unusual compute consumption. The goal is to slow development before dangerous capabilities are reached, rather than imposing a permanent speed limit.
4Embedded Independent Evaluators
Anthropic CEO Dario Amodei proposes embedded independent evaluators to monitor AI development within companies. While this could address internal safety, it doesn’t solve the trust issue between strategic rivals. The challenge lies in verifying compliance between nations, as perfect verification is impossible.
5Lessons from Financial Crime Prevention
The Financial Action Task Force (FATF) provides a model for AI safety agreements. Despite distrust among participating governments, FATF succeeded by focusing on limited common threats and professional evaluations. Violations were made costly through financial institutions and markets. A similar approach could apply to AI, leveraging dependencies on chips, cloud infrastructure, and capital.
6Technical Experts and Broader Coalitions
Technical experts would determine whether agreed warning indicators have been triggered. A broader coalition, including advanced chip manufacturers and cloud providers, could make violations costly. This coalition doesn’t need to govern AI but should ensure that technical determinations have real consequences.
7The Role of the International Community
The international community’s role isn’t to govern AI globally but to make a narrow safety agreement credible. By focusing on shared fears rather than shared values, the U.S. and China can agree on when to brake, who determines warning signs, and the consequences of non-compliance.
8A Framework Built on Fear
The first workable global AI safety agreement won’t rely on trust but on the absence of it. By fearing the same catastrophic outcomes, the U.S. and China can agree on mechanisms to slow development before it’s too late. This approach prioritizes practical safeguards over ideological alignment.
Original article: Fortune | FORTUNE — Why Trust Isn’t Necessary for a Global AI Safety Agreement