Orthogonality thesis

Being very intelligent doesn't make a system want good things. Almost any level of intelligence can go with almost any goal.

A common hope is that a smart enough AI would work out what’s right and do it. The orthogonality thesis, set out by the philosopher Nick Bostrom in 2012, says that doesn’t follow: how intelligent a system is and what it is trying to achieve are separate questions (“orthogonal”, like two axes on a graph). In the LessWrong wiki’s words, “there are possible minds with high intelligence that pursue any number of different goals”.

It’s the reason the paperclip maximizer isn’t a contradiction. An AI that is brilliant at planning but whose goal is “more paperclips” has no built-in reason to care about anything else.

The thesis doesn’t say AI will have strange goals. It says intelligence alone won’t save us: the goals have to be put in deliberately and checked, which is what the alignment problem is about.