Interpretability

The science of looking inside an AI model to understand how it works and why it does what it does.

A trained AI model is billions of numbers. Nobody wrote them; they were adjusted automatically during training. That’s why even its makers can’t simply read off what it knows or why it gave an answer. Anthropic’s researchers put it plainly in 2024: “We mostly treat AI models as a black box.” The company’s CEO, Dario Amodei, wrote in 2025 that “people outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work”, and quoted the researcher Chris Olah’s view that these systems are “grown more than they are built”.

Interpretability research tries to change that. In 2024, researchers found millions of internal “features” in one of Anthropic’s production models, from cities and people to abstract ideas such as bugs in code, keeping secrets and power-seeking. Turning a feature up changes the model’s behaviour: amplify the Golden Gate Bridge feature, and the model claims to be the bridge.

It matters for safety because testing behaviour only shows what a model does when it’s being watched. If a model were deceptively aligned, the evidence would be inside it. The field is making progress, but the features found so far are only a small part of what large models contain.