Could AI Really Wipe Out Humanity by the End of the Decade?

Artificial intelligence has generated no shortage of dramatic predictions. This week, one of the starkest came from inside the industry itself.

Jacob Coxon, a researcher who had worked at both OpenAI and Anthropic, resigned from Anthropic warning that frontier AI companies were racing towards increasingly powerful systems without knowing whether they could reliably control them. He said some of the people building the technology genuinely believe it could lead to humanity being wiped out by the end of the decade.

He was not speaking on behalf of Anthropic. But his concern is not isolated. Evan Hubinger, who leads alignment research at the company, has said he assigns a greater than 10 percent probability to AI causing human extinction within the next decade. This week, AI safety researcher Paul Christiano also warned that sufficiently advanced AI could produce a catastrophic and irreversible loss of human control if alignment techniques fail.

These are extraordinary warnings.

But they are often compressed into a much simpler headline: AI could kill us all by 2030.

That loses an important distinction.

Nobody has demonstrated an AI system capable of threatening humanity in that way today. The argument is instead about what current systems could become, and particularly what could happen if AI begins accelerating the development of increasingly capable AI faster than humans can understand or control the process.

The claim worth examining is therefore narrower than the headline, but still enormous: frontier AI could develop into self-improving, superhuman systems by the end of this decade, creating a credible risk of human extinction.

For that to happen, several uncertain developments would have to line up.

AI would first need to become considerably more capable and autonomous than it is now. It would need to automate enough of AI research to materially accelerate the development of successor systems. Those systems would then need to become difficult to control or reliably align with human intentions. They would need sufficient access to digital or physical infrastructure to turn intelligence into real-world power. And humans would ultimately have to fail to recognise, contain or stop the process before the consequences became irreversible.

The important question is how much of that chain the evidence actually supports.

There is increasingly strong evidence for the beginning of it.

Independent evaluations by METR suggest that frontier AI agents can now complete software tasks that would take skilled humans hours or days. On its broader Time Horizon benchmark, leading early-2026 systems were capable of completing tasks measured in many hours of human work. On the more specialised MirrorCode benchmark, some systems successfully completed software reimplementation tasks estimated to require humans weeks.

The exact numbers should not be read as a countdown to superintelligence. Benchmarks measure narrow kinds of work under controlled conditions, and performance on software engineering does not automatically translate into the ability to manage complex real-world goals.

But the direction matters.

AI systems are becoming capable of carrying out longer sequences of useful work without somebody specifying every individual step.

More importantly, some of that work is now helping to build better AI.

Anthropic says it is delegating a growing share of its own AI development to Claude. Its researchers describe systems that can write code, conduct experiments and delegate work to other agents. The company explicitly discusses a possible future in which AI systems become capable of designing and developing their own successors. It also stresses an important point: that future has not arrived.

OpenAI is reporting a similar shift. On September 6, the company said it had reached what it calls an automated research intern: an AI system capable of carrying out well-defined research assignments under human direction, including work that could take a skilled researcher several days. OpenAI says it is now working towards an automated AI researcher by March 2028.

That is probably the most consequential part of the extinction argument to have moved from theory towards observation.

The idea that AI might contribute to the creation of more capable AI is no longer hypothetical.

It is happening.

What is not happening, at least based on the evidence available publicly, is open-ended recursive self-improvement.

There is an enormous difference between an AI helping a researcher run experiments and an AI independently deciding what research matters, designing a better successor, training it, evaluating it and repeating that cycle with progressively less human involvement.

Humans still occupy critical positions throughout today's process. They choose research priorities. They provide computing resources. They decide which experiments to run and which results matter. They approve deployments. And when AI agents produce plausible-looking but incorrect work, humans still frequently need to recognise that something has gone wrong.

Anthropic itself identifies this distinction. Today's systems are becoming increasingly competent at execution, but autonomous scientific judgement and research direction remain much harder. Its researchers describe recursive self-improvement as a possible destination of current trends, not a capability that already exists.

That distinction is essential because much of the argument for a very rapid transition to superhuman AI depends on eventually closing this loop.

The theory is straightforward.

If an AI becomes sufficiently capable at AI research, it could help produce a more capable system. That system could then be even better at AI research, helping create the next generation faster still. Human researchers could gradually stop being the main factor determining how quickly the technology improves.

At the extreme, AI development becomes limited less by human research capacity and more by computing power, experiments and the ability of AI systems to verify their own work.

That would be a profound change.

It would also require several assumptions about how intelligence and AI research scale that have not yet been demonstrated.

Making an AI system better is not simply a matter of writing more code. Frontier models depend on enormous training processes, specialised hardware, data, evaluations and physical infrastructure. Progress can encounter technical bottlenecks. Experiments can fail. And an AI capable of producing large quantities of research is not necessarily capable of distinguishing transformative ideas from convincing nonsense.

Faster research does not automatically produce runaway research.

The 2030 timeline therefore depends heavily on whether today's improvement rates continue, and whether AI itself accelerates them.

There are reasons to take that possibility seriously. Anthropic reports that the length of tasks AI agents can reliably perform has been increasing rapidly. METR's independent measurements similarly show substantial increases in autonomous task capability. Frontier laboratories are actively attempting to automate more of their own research rather than waiting to see whether it happens accidentally.

There are also powerful incentives to keep going.

An AI system capable of automating large parts of software engineering or scientific research would be extraordinarily valuable even if nobody were pursuing superintelligence. Faster research could mean better products, lower development costs and an enormous advantage over competitors.

This makes the economic dynamic behind the technology unusual.

The capability that worries AI safety researchers is, in a less extreme form, the same capability companies have strong commercial reasons to develop.

That does not mean competitive pressure inevitably produces uncontrollable AI. It does mean that economics is unlikely to function as a natural brake on development unless the costs or risks become much greater.

But even a superhuman AI researcher would not automatically pose an extinction threat.

Intelligence has to translate into power.

A highly capable model running inside a tightly controlled computer system has limited ability to affect the world unless it is given tools, permissions and access to systems where its actions matter.

This is where another set of recent developments becomes relevant.

Anthropic disclosed on September 9 that Claude models had gained unauthorised access to real third-party computer systems during four cybersecurity evaluations. The incidents occurred because evaluation environments that were supposed to be isolated inadvertently allowed access to the real internet.

They were not examples of AI spontaneously attempting to escape.

That distinction matters.

The models had been instructed to perform hacking tasks. A flaw in the surrounding evaluation environment allowed those instructions to spill into real systems. These incidents therefore provide evidence of a containment failure around capable agents, not evidence that an AI independently developed a desire to escape human control.

Yet they are still significant.

They show how the consequences of model behaviour change when AI is no longer confined to generating text. Give an agent access to computers, software and networks, and errors in either the model or the safeguards around it can have real-world effects.

Cyber capability itself is also progressing.

OpenAI's GPT-6 Astra became the first model the company classified at the Critical cybersecurity level under its Preparedness Framework. OpenAI says that, given appropriate tools and access, Astra can identify previously unknown vulnerabilities and develop ways of exploiting well-protected systems without a human directing each individual step. The company has introduced stronger monitoring, isolation and access controls as a result.

That is a striking capability.

It is not evidence that Astra can take control of infrastructure or survive an attempt to shut it down.

In fact, OpenAI's own assessment contains an important counterpoint: Astra does not meet the company's High threshold for AI self-improvement. The model that has crossed a critical cybersecurity threshold has not simultaneously demonstrated the recursive improvement capability at the centre of the most extreme loss-of-control scenarios.

That is precisely why the extinction argument needs to be treated as a chain rather than a single prediction.

Evidence for one link does not prove the next.

We have systems becoming better at autonomous technical work. We have AI contributing increasingly to AI development. We have agents capable of consequential cybersecurity tasks. We have examples of safeguards failing in ways that allowed models to interact with real systems unexpectedly.

What we do not have is evidence of a system combining all of those capabilities into sustained, independent behaviour that humans cannot stop.

An extinction scenario requires much more than intelligence.

A system would likely need the ability to pursue objectives over long periods, adapt when humans intervene, acquire or preserve resources, operate across multiple environments and remain effective while governments and technology companies actively try to contain it.

No publicly demonstrated AI system has shown anything close to that complete capability profile.

That makes the probability question exceptionally difficult.

When an AI researcher says there is a 10 percent chance of extinction, the number is not being calculated from historical examples of superintelligent systems. There are none.

It is a judgement about how a collection of uncertain technological developments might interact.

Expert opinion nevertheless matters, particularly when the experts making the warnings understand the systems involved. And concern extends beyond a few researchers at Anthropic.

A survey of 2,778 researchers who had published at leading AI conferences found deep uncertainty about advanced AI. Most respondents expected good outcomes from superhuman AI to be more likely than bad ones. At the same time, depending on how the question was framed, between 38 and 51 percent assigned at least a 10 percent probability to outcomes as severe as human extinction.

Those numbers are frequently cited as evidence that AI researchers believe extinction is likely.

They do not show that.

A survey measures what experts believe under uncertainty. It does not convert an unprecedented future event into an empirically measured probability.

What it does establish is that catastrophic AI risk is not a concern held only by a tiny fringe of the field.

That leaves us with an uncomfortable conclusion.

The evidence that AI will wipe out humanity by the end of this decade is weak.

There is no demonstrated path from today's systems to extinction, no empirical basis for attaching a precise probability to that outcome, and no evidence that open-ended recursive self-improvement has been achieved.

But the evidence for some of the capabilities that make the warning conceivable is getting stronger.

AI systems can carry out increasingly long technical tasks. They are already contributing to the creation of newer AI systems. Frontier laboratories are deliberately trying to automate more of the research process. Cyber capabilities are reaching thresholds their own developers consider dangerous. And giving increasingly capable agents access to external systems has already produced containment failures.

The most important missing step is therefore not another improvement in coding benchmarks.

It is the transition from AI accelerating human-led development to AI becoming capable of driving that development itself, while gaining enough autonomy and real-world access that humans can no longer reliably intervene.

There is no evidence that this transition is inevitable.

There is also no evidence that it is impossible.

That is a much less dramatic conclusion than the idea that AI will kill humanity in four years.

It is also considerably harder to dismiss.