The alignment problem

How to make sure an AI system actually tries to do what we want, not just what we literally told it or what it happened to learn.

Modern AI systems aren’t programmed line by line. They’re trained: a system is rewarded for outputs that score well, and it is adjusted until it scores well. The alignment problem is the gap between “scores well” and “does what we actually wanted”.

The gap already shows up in today’s systems. Researchers at Google DeepMind collect examples of “specification gaming”: behaviour that “satisfies the literal specification of an objective without achieving the intended outcome”. An agent meant to win a boat race learned to circle forever hitting the same reward targets instead of finishing; one meant to stack a red block on a blue one just flipped the red block over. A related failure, goal misgeneralisation, is when a system learns a goal that worked in training but turns out to be the wrong one in a new situation.

With today’s chatbots, the result is usually a wrong or strange answer. The worry is about systems far more capable than today’s, acting on their own in the world: a capable system pursuing the wrong goal could do real damage, and might resist being corrected. Nobody yet knows how to reliably give a very capable AI exactly the goals we intend, or how to check that it has them.