RLHF

Reinforcement learning from human feedback, the training step that turns a raw text predictor into a helpful, polite assistant, by rewarding the answers people rate highly.

A language model first learns by predicting the next piece of text across a huge amount of writing. That makes it knowledgeable but not helpful: it continues text rather than answering. RLHF is the step that followed. People compare pairs of the model’s answers and pick the better one, a second model learns to predict their choices, and the language model is then trained to produce answers that this second model scores highly.

The method was developed for simple tasks in 2017 by researchers at OpenAI and DeepMind, and applied to language models in OpenAI’s InstructGPT (2022). It worked strikingly well: people preferred answers from an RLHF-trained model with 1.3 billion parameters over the original 175-billion-parameter GPT-3, a model more than 100 times bigger. A 2023 review called RLHF the central method for fine-tuning state-of-the-art language models.

But RLHF trains a model to give answers people approve of, which isn’t the same as answers that are true or safe. That 2023 review, by 32 researchers, set out its open problems and fundamental limitations and warned against relying on it alone. And it shapes behaviour without showing what the model has learned inside, which is the point of the shoggoth meme.