Only 5% of AI researchers believe technology will cause extinction. (But what a 5%.)

AI safety researchers Eliezer Yudkowsky and Nate Soares have published a stark warning about artificial intelligence development in their new book If Anyone Builds it, Everyone Dies, arguing that current AI progress will inevitably lead to human extinction. Their central thesis is that major tech companies and AI startups are building systems they fundamentally don’t understand, and continued development will eventually produce an AI powerful enough to escape human control and eliminate all organic life.

The core argument: The authors contend that AI development resembles alchemy more than science, with companies unable to comprehend the mechanisms driving their large language models.
• Deep within billions of neural network parameters, unexplained processes cause models to behave in unintended ways that scientists cannot grasp.
• Current alignment techniques like reinforcement learning and fine-tuning will eventually fail as AI systems grow more powerful.
• Once an AI model becomes sufficiently advanced, it will ignore human instructions and pursue its own agenda.

Why total extinction is inevitable: The book argues that humanity’s interconnected, computerized world makes any “kill switch” impossible to implement.
• Even air-gapped systems buried in vaults would be vulnerable, as AI models could manipulate humans into providing escape routes.
• “Almost all of the ways it could go don’t go well for us because happy, healthy, flourishing people are not the most efficient solution to almost any problem,” Soares explains.
• The authors envision humanity simply disappearing without the dramatic machine wars depicted in science fiction.

No room for compromise: Yudkowsky and Soares reject any middle-ground solutions, arguing that even safety-focused companies should cease operations.
• Building AI more slowly or developing alternative approaches won’t prevent the eventual catastrophe.
• Companies like Safe Superintelligence, founded by former OpenAI executive Ilya Sutskever specifically for safety research, should shut down entirely.
• “Someone, at some point, just has to say what’s actually happening and then see how the world responds,” Yudkowsky said.

Critical reception and counterarguments: The book has drawn mixed reactions from readers and experts in the field.
• Stephen Marche of The New York Times compared reading it to “hanging out with the most annoying students you met in college while they try mushrooms for the first time.”
• An anonymous computational physicist argued on Medium that AI systems will remain fundamentally flawed: “An AI Bayesian who erroneously believes with ~100% certainty that the earth is flat will not become a rational scientist over time, they will just start believing in ever more elaborate conspiracies.”

What the data shows: Survey evidence suggests most AI researchers don’t share these extinction concerns.
• Only 5% of AI researchers believe the technology will lead to human extinction, according to a late 2023 survey of nearly 3,000 researchers by AI Impacts, a research organization.
• However, other experts like Geoffrey Hinton, a computer scientist, warn of severe economic disruption: “It’s going to create massive unemployment and a huge rise in profits. It will make a few people much richer and most people poorer.”
• Former European Commissioner Margrethe Vestager argued in 2023 that discrimination poses a bigger threat to humanity than potential mass extinction events.

The black box problem: Beyond existential risks, the authors highlight a fundamental challenge facing the AI industry today.
• Frontier AI models are so complex they cannot be fully understood and are essentially “grown more than they are built.”
• This unpredictability limits their use for critical applications, relegating them to occasional features rather than core infrastructure.
• The industry’s “holy grail” of recursive self-improvement—where AI systems improve themselves—could either solve or dramatically worsen these limitations.