The Static Target Trap
Treating human values as a fixed object to be found and encoded misses that they are dynamically maintained, socially mediated, and constantly revised.
What decision changes?
Do not ask a system to maximize a snapshot of human values. Ask it to preserve and assist the legitimate human value-update process.
The familiar alignment picture is: there is an artificial system, there are human values, and the task is to make the system optimize those values. That picture explains real dangers — a powerful optimizer aimed at the wrong target, or a proxy that breaks outside its training distribution — but it hides a deeper difficulty.
Human values are not fixed objects, preferences, reward functions, or verbal principles. They are dynamically maintained patterns in biological, cognitive, social, and institutional systems: compressed summaries of error signals, needs, and coordination pressures, revised through evidence, argument, education, law, and technology.
This matters because superintelligence does not merely act in a world containing human values — it changes the world in which those values are formed: the evidence people see, the incentives they face, the institutions they trust, and eventually perhaps the substrate their agency runs on. A target that only maximizes today’s snapshot cannot survive that.