Existential risk

The risk that advanced AI causes human extinction, or a similarly permanent loss of humanity's future.

An existential risk is one that could end humanity or permanently wreck its future. The AI Safety Atlas treats it as the most severe end of a range of AI harms, and the scenarios it covers include human extinction, humanity permanently losing power over its own future, and the collapse of civilisation.

It isn’t a fringe worry. In 2023, hundreds of AI researchers and the heads of OpenAI, Google DeepMind and Anthropic signed a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” In the largest survey of AI researchers so far (2,778 people who publish at top AI conferences), between 38% and 51% gave at least a 10% chance that advanced AI leads to outcomes as bad as human extinction.

There are several roads there. The Atlas sorts them by cause: misuse (people using AI to do harm), misalignment (AI pursuing goals nobody intended), and systemic risks that build up across society, made worse by “risk amplifiers” such as the race between companies and countries. The common thread is that the damage might not be fixable afterwards, so it has to be prevented in advance.