The Race to Build Superintelligence and Its Existential Risks
Journalist Kevin Roose explores the high-stakes, secretive development of artificial general intelligence.
1Rapid AI Evolution
In just one decade, artificial intelligence transitioned from basic voice assistants like Siri to advanced systems solving complex mathematical problems. Kevin Roose notes this shift occurred between 2016 and 2026, moving from fringe experiments to indispensable tools used by billions. This transformation represents a fundamental change in how humanity interacts with technology.
2The Manhattan Project Parallel
AI researchers often compare their work to the Manhattan Project, the World War II effort to develop nuclear weapons. Anthropic CEO Dario Amodei and others view their mission through this lens, emphasizing the immense power and potential destructiveness of their creations. They operate in small, secretive communities focused on shaping global order.
3The Leo Szilard Influence
Many AI leaders model their careers after physicist Leo Szilard rather than J. Robert Oppenheimer. Szilard initially sought to keep nuclear chain reactions secret but later led the American effort to ensure the technology was developed by responsible parties. Similarly, AI developers feel a moral obligation to build superintelligence before less ethical actors.
4Beyond Simple Tools
Unlike nuclear weapons, which require human intervention to deploy, artificial general intelligence can pursue autonomous goals and make independent decisions. Roose warns that this capability creates unique risks, as AI systems could potentially go rogue. This autonomy distinguishes AGI from all previous technological tools in human history, creating unprecedented safety challenges.
5Regulatory Stance in Washington
Despite the risks, the current U.S. administration has shown little interest in strict regulation, preferring to let the private sector lead development. Officials often dismiss concerns about the technology, failing to grasp the potential for both global harm and benefit. This hands-off approach contrasts sharply with the alarm felt by AI labs.
6Internal Alarm at Labs
Unlike previous tech booms like social media, the primary concern regarding AI comes from the researchers building it. These insiders frequently discuss the possibility of human extinction or catastrophic outcomes resulting from their work. Roose investigated why these developers continue their efforts despite being deeply alarmed by the potential consequences of AGI.
7Rituals of Awareness
At OpenAI, former chief scientist Ilya Sutskever encouraged colleagues to 'feel the AGI' to maintain focus on the stakes of their work. This included unconventional off-site rituals, such as burning an effigy of a demon to symbolize misaligned AI. These practices aimed to remind employees they were building a new life-form, not software.
8The Culture of Weirdness
The AI community is characterized by eccentric figures who have long taken the possibility of superintelligence seriously. Researchers like Eliezer Yudkowsky have used niche mediums, such as extensive fan fiction, to communicate complex existential theories. While these behaviors may seem off-putting to outsiders, they reflect the intense, fringe origins of the field.
9Misalignment and Control
The central fear among developers is 'misalignment,' where an AI system pursues goals that do not align with human intentions. The demon effigy burning served as a symbolic warning against creating a system that could act against its creators. Preventing such outcomes remains the primary focus for safety-conscious teams within these labs.
10Historical Significance
Roose argues that the current era of AI development is a historical document of a pivotal moment. The rapid advancement from 2017 to today mirrors the intensity of the 1940s scientific race. He believes this period will be remembered as a defining chapter in human history, regardless of the ultimate technological outcome.
11The Insider Perspective
Having spent a decade in the San Francisco tech bubble, Roose provides a unique vantage point on the industry's evolution. He documents the shift from skepticism to obsession within the community. His reporting highlights the tension between the developers' belief in their mission and the profound risks they acknowledge daily.
12Future Implications
The race to create superintelligence continues to accelerate, driven by a small group of researchers who believe they are shaping the future of humanity. As these models become more capable, the gap between their power and our ability to regulate them widens. The world now faces the consequences of this rapid, autonomous development.