The Race to Build Artificial Superintelligence Accelerates
Journalist Kevin Roose explores the high-stakes, secretive development of autonomous AI systems.
1A Decade of Transformation
Between 2016 and 2026, artificial intelligence evolved from basic voice assistants like Siri to advanced systems capable of solving complex mathematical problems. This rapid shift transformed AI from a niche experimental tool into an indispensable technology used by billions. Kevin Roose documents this historical acceleration in his latest book, The AGI Chronicles.
2The Manhattan Project Parallel
AI researchers frequently compare their work to the Manhattan Project, the World War II effort to develop nuclear weapons. Anthropic CEO Dario Amodei and his team studied the project to understand the ethical weight of their creation. Like the atomic bomb, they view AGI as a potentially world-altering, destructive, and transformative technology.
3The Leo Szilard Model
Dario Amodei modeled his career after physicist Leo Szilard rather than J. Robert Oppenheimer. Szilard initially attempted to keep nuclear research secret to prevent misuse by adversaries. When that failed, he pushed for the United States to build the technology first, believing it was a moral obligation to prevent bad actors from succeeding.
4Tools Versus Autonomous Agents
Unlike nuclear weapons, which require human intervention to deploy, AGI possesses the capacity for autonomous action and goal pursuit. Roose argues this distinction is critical, as AI could potentially go rogue. These systems might execute plans that humans do not understand, creating risks that extend far beyond traditional tool-based technological threats.
5Regulatory Stance in Washington
Despite the existential risks identified by researchers, the current U.S. administration shows little appetite for strict regulation. Officials prefer to let the private sector lead development, often described as letting the industry cook. Roose suggests that policymakers may not fully grasp the implications of granting global access to the most advanced models.
6Insiders Expressing Alarm
Unlike previous tech booms, the most vocal critics of AI are the very researchers building it. While social media executives rarely claimed their products could cause human extinction, AI lab employees frequently express deep concern. Roose’s reporting aims to uncover why these specific individuals remain alarmed while continuing to develop more capable systems.
7Rituals of AGI Awareness
At OpenAI, former chief scientist Ilya Sutskever encouraged colleagues to feel the AGI to maintain focus on the stakes. This included unconventional off-site rituals, such as burning a wooden effigy of a demon to symbolize misaligned AI. These practices were intended to remind staff they were building a new, potentially dangerous life-form.
8The Feel the AGI Day
OpenAI continues to hold versions of these awareness exercises, now known as Feel the AGI Day. While they have moved away from burning effigies, the goal remains to impress upon employees the gravity of their work. The company seeks to ensure that safety remains a priority during the rapid development of new models.
9Juxtaposition of Weirdness
The AI community often displays a strange blend of high-level scientific rigor and eccentric behavior. Researchers like Eliezer Yudkowsky have used long-form fan fiction to communicate complex existential risks. Roose notes that one had to be considered weird to take AGI seriously in 2017, as the technology was not yet mainstream.
10Historical Significance of Research
Roose views his book as a time capsule rather than a minute-by-minute update on AI progress. He focuses on the human experience of the researchers who transitioned from building simple software to creating systems that change the global order. He believes this era will be remembered as a pivotal moment in human history.
11The Challenge of Regulation
Regulating AGI is difficult because the technology is evolving faster than public or political understanding. Many people do not grasp the underlying science, making it hard to implement effective guardrails. Roose suggests that the lack of public discourse mirrors the confusion that followed the first use of the atomic bomb in 1945.
12Future Implications and Outlook
The race to build superintelligence continues to accelerate, driven by a small, secretive community of researchers. While the potential for good is immense, the risk of catastrophic misalignment remains a central concern for the creators themselves. The coming years will determine whether these systems can be safely integrated into the global human order.
Original article: Scientific American — The Race to Build Artificial Superintelligence Accelerates