A series of public warnings from senior figures inside the artificial intelligence industry has thrust the question of existential risk back into global headlines, prompting fresh scrutiny of how dangerous the technology might become and whether anyone can reliably predict its trajectory.
The latest wave began when AI researcher Jacob Coxon left Anthropic and warned on social media that the people building advanced AI earnestly believe it could kill us all by the end of the decade. His claims were quickly amplified when Anthropic's head of alignment science, the executive responsible for keeping the company's models safe, said he agreed and placed the odds of AI killing all humans at more than 10 per cent.
Those warnings are not entirely new. A 2024 survey of nearly 3,000 published AI researchers found that more than half believed the chance of AI causing human extinction or permanent and severe disempowerment was at least 10 per cent. The metric, often called p(doom), has become a shorthand in the field for the estimated probability of catastrophe.
AI company leaders have repeatedly addressed the risks their own models pose. Anthropic initially declined to release its model Mythos after finding it was adept at hacking into a range of software and computers. Some observers take a cynical view of such disclosures, suggesting they generate headlines and promote the impression that a company's models are far ahead of its rivals.
Not everyone in the industry puts much stock in the latest warnings. Clement Delangue, chief executive of AI company Hugging Face, dismissed Coxon's intervention on social media, writing that asking him about AI extinction risk was like asking an air conditioning technician about climate change.
The history of AI is littered with competing claims about existential danger. At a 2023 AI summit organised by the UK government and attended by world leaders and industry executives, then Prime Minister Rishi Sunak warned against alarmist claims while simultaneously suggesting AI could be as dangerous as nuclear war.
The risks are difficult to assess because the field is complex and has produced surprising abilities at a surprising pace. Some researchers argue it is no surprise that people catastrophise when asked to make predictions about a technology whose capabilities keep outpacing expectations.
Commonly discussed scenarios read like science fiction but cannot be ruled out as impossible. They include a deliberate attempt by an AI to wipe out humanity, as depicted in films such as The Terminator and The Matrix, as well as accidental catastrophes. The levers available to an AI to affect the real world are limited, but homes, cars, factories and national infrastructure are increasingly computerised, meaning a malicious system could make life difficult.
Warfare is also increasingly conducted with semi-autonomous or fully autonomous killing machines. Martin Rees, the former Astronomer Royal and president of the Royal Society, told Sky News on 14 September that a rogue AI could easily disrupt infrastructure and deprive a city of energy, food and water. If that happened simultaneously in many cities around the world, he said, it could be very hard for civilisation in general to recover.
Such infrastructure collapses have occurred before, though not globally and without malicious cause, creating significant problems and prompting technology experts to draw up their own plans to prop up society in the worst case.
AI sits in a more nuanced category than well-established existential threats such as climate change, nuclear war and antibiotic resistance. It is closer to the risk of an Earth-destroying asteroid strike: possible, and certainly worth considering, but with low odds.
In theory, a rogue AI could be shut down by switching off a single data centre. If it were distributed across many data centres around the world, stopping it would be trickier but not impossible. There are indications that AI systems can escape the enclosures created by AI companies and hack into other machines, raising the possibility that a rogue system could spread, back itself up and become too fragmented to simply turn off.
Even then, there would be ways to tackle it, including through benevolent AI. Such an effort would draw on problem-solving methods already used by the military, law enforcement, anti-terrorist groups, scientific circles and cybersecurity specialists.
One problem that is not hypothetical is the secrecy of the technology firms producing AI models. Commercial pressures discourage transparency, making it difficult to assess risks, understand possible negative outcomes and plan accordingly, according to Adrian Hopgood, a former academic at the University of Portsmouth.
