Recursive self-improvement

Programmers, programmer, developer, software engineer, coder, coding, IT, Scientists and researchers, scientist, researcher, research, academic, laboratory, lab

AI that helps build better AI, which builds better AI still, in a feedback loop that could speed progress up dramatically.

Most technologies don’t improve themselves. AI might: designing and training AI systems is itself intellectual work, and if AI gets good at that work, each generation could help build a stronger next one. The AI Safety Atlas describes this as “an AI system becomes better at improving itself”, a feedback loop that keeps speeding up. The idea goes back to the statistician I. J. Good, who argued in 1965 that a machine smarter than us could design even better machines, leading to an “intelligence explosion”.

How fast such a loop would run is disputed. In the “fast takeoff” picture it runs away quickly, leaving little time to react. In the “slow takeoff” picture progress compounds more gradually, though by one estimate still 10 to 100 times faster than the Industrial Revolution. The Atlas notes that the difference often comes down to whether problems would be noticeable and fixable in time. Scenarios such as AI 2027 treat AI doing AI research as the turning point.

It matters for safety because a fast loop shrinks the time to notice and fix problems. The EU’s Code of Practice for the most capable AI models lists “self-improvement” and AI “autonomously creating or improving AI models” among the ways humans could lose control.