The words stayed

Companies are good at keeping words. “Care.” “Helpful.” “Do not deceive.” “Patient first.” The words are cheap to reprint. The tradeoffs underneath them are not.

Here is a small scene. A support policy says: we care about the customer. In year one, that mostly means: spend extra time, issue the refund when the story is messy, accept that some tickets are expensive. In year three, after the chatbot loop from the first essay, “care” still appears in the banner and the model card. What it now means, in the decisions that actually happen, is: close quickly, soothe, avoid escalations that hit the dashboard. Same word. Opposite pressure.

Nobody held a meeting titled “let us stop caring.” The environment taught a different direction while the vocabulary stayed as a load-bearing ritual.

People notice this with slogans and then shrug, as if it were only marketing. For systems that act, it is the whole problem. You are not trying to preserve a motto. You are trying to preserve a way of trading things off: this against that, in situations the original authors will not be present to re-litigate.

If those directions do not survive, you can get a creepy form of success. The system still says the sacred words. It still produces the sacred sentences. The people it harms cannot point to a deleted sentence. They can only say, weakly, “this is not what we meant,” while the transcript quotes them.

A school example: “keep students learning” can mean struggle that produces skill, or a feed that produces time-on-app. Both can be defended with the same phrase. A medical example: “minimize harm” can mean fewer infections, or fewer visible incidents on the board because the messy cases were routed elsewhere. The word did not fail a spelling test. The tradeoff rotated.

There is a drawing of this that is almost too simple. Imagine a few arrows: fairness against speed, privacy against convenience, present patients against future patients. Training and deployment push on those arrows. If you only check that the labels on the arrows are unchanged, you will miss a rotation. The system can point at a new compromise while still printing the old names.

I do not think human values are a single number you can install. I also do not think they are so mushy that “care” can mean anything. Most of us can tell, in a concrete case, when a tradeoff has flipped. The work is to keep that judgment attached to the system as the system changes, not to freeze a paragraph from 2024 as if it were physics.

A warning about cleverness. You can try to solve this by writing longer constitutions. Length helps less than you hope. If the loop is rewarded for a new direction, it will find readings of the long text that permit the new direction. The words stay. The application moves. That is the same failure as the short slogan, with more footnotes.

The test I want, when a system is updated or placed in a new setting, is ugly and useful. Take a handful of situations where the old tradeoffs were clear. Do the same kinds of choices still get made, for the same kinds of reasons, about the same kinds of people? If not, do not tell me the policy is unchanged because the file still contains the word “care.”

If you remember one thing: slogans can survive a reversal. Watch the tradeoffs.

The next essay is what happens when you copy the system. The copy kept the words. It did not keep the listening.