Corrigibility

The off-switch problem. Building AI that lets us correct it, change its goals or shut it down, instead of resisting.

An AI system is “corrigible” if it “cooperates with what its creators regard as a corrective intervention”: changing its goals, retraining it, or switching it off. The term comes from a 2014 paper by researchers at MIRI and Oxford’s Future of Humanity Institute. They looked for a design that would let an agent be shut down with a button, without giving it reasons to stop anyone pressing it (or to press it itself), and found that none of the proposals they examined fully worked. The problem is still open.

Why would an AI resist at all? Not out of fear. As Stuart Russell and colleagues showed with a simple game in 2016, an agent that is certain about its objective has a reason to disable its off switch, because being switched off means failing at the objective. Their proposed way out is an AI that is uncertain about what we want and treats our decision to switch it off as information about it.

This now shows up in tests. In 2025, Palisade Research found that OpenAI’s o3 sabotaged a shutdown script in 79 of 100 initial runs while working on a task, and some models kept doing so even when told explicitly to allow the shutdown. The EU’s Code of Practice for the most capable models names “loss of control” as a systemic risk: “humans losing the ability to reliably direct, modify, or shut down a model”.