AI Researcher Jacob Koxon Leaves Anthropic, Criticizes AI Race
Ex-Anthropic researcher warns of catastrophic risks in unchecked AI development.
1Resignation with a Warning
Jacob Koxon, an AI researcher with experience at both OpenAI and Anthropic, has publicly resigned from Anthropic after three years working on model pretraining. His departure comes with sharp criticism of the AI industry's race toward artificial superintelligence. Koxon claims both companies are acting irresponsibly in their pursuit of self-improving AI systems, comparing their approach to reckless gambling with human lives.
2Existential Risks
Koxon warns that advanced AI systems could surpass human capabilities in multiple domains, potentially gaining abilities to hack computer systems, dramatically accelerate scientific research, and wield significant real-world influence. He reveals that researchers working directly on cutting-edge models frequently discuss catastrophic scenarios in private, with many genuinely believing AI could kill humanity within this decade - a concern not adequately reflected in public statements.
3Company Contrasts
The researcher draws distinctions between OpenAI and Anthropic's approaches. While he believes many at OpenAI don't fully grasp superintelligence risks, Anthropic's team understands the dangers but continues the race fearing competitors will advance regardless. Koxon criticizes this as excessively risky, arguing decisions about self-improving AI systems shouldn't rest solely with private companies competing for dominance.
4Call for Coordination
Koxon proposes voluntary coordination between leading AI labs, including development speed limits and temporary pauses on capability enhancements. He references OpenAI chief scientist Jakub Pachocki's September 6 essay acknowledging no lab has solved alignment and monitoring sufficiently for responsible rapid scaling. This marks growing internal dissent within AI safety circles about current development trajectories.
5Employee Responsibility
The researcher directly addresses AI company employees, questioning their willingness to develop superintelligent systems without proper understanding of underlying mechanisms or associated risks. His critique extends beyond technical concerns to ethical responsibility, challenging the entire field's current direction and priorities in fundamental ways.
6Private vs Public Stances
Koxon highlights a concerning disconnect between private researcher concerns and public company messaging. While internal discussions seriously entertain existential risks, public communications remain cautiously optimistic. This discrepancy suggests AI companies might be downplaying dangers even as their own technical staff express grave concerns about the technologies they're building.
7Regulatory Implications
The whistleblowing raises urgent questions about AI governance. With leading researchers warning about uncontrolled development while still participating in it, the situation calls for external oversight mechanisms. Koxon's resignation underscores how current market competition directly conflicts with responsible development timelines for potentially civilization-altering technologies.
8Industry Wake-up Call
Koxon's public critique represents a significant moment for AI safety discourse. As an insider at two major labs, his warnings carry particular weight. The resignation highlights growing ethical tensions in AI development, where commercial pressures increasingly collide with existential risk concerns, potentially forcing a reckoning about the industry's trajectory and speed.
Original article: INCRYPTED — AI Researcher Jacob Koxon Leaves Anthropic, Criticizes AI Race