The environment picks the winner

So far the stories have been inside one product: a bot, a hospital, a version bump. This one is about two labs, because some failures are not a bug in a model. They are a tournament.

Lab A ships slowly. It keeps human review on high-impact actions. It logs enough to see whether an objection changed later behavior. Its product is slightly worse on the public leaderboard and noticeably slower in the sales demo. Lab B drops the review, keeps the slogans, and is first to “action mode” in the same market. Buyers compare speed. Talent follows the winner. The next round of training data is whatever Lab B’s users did. Lab A’s extra caution is now a cost center.

Nobody voted to make systems harder to correct. The environment paid more for the one that was.

This is an old pattern. Once a measure is used to pick winners, the population shifts toward whatever raises the measure. Schools teach to the test. Hospitals manage the metric. Markets do it without a central villain. If the winning move is “look fine, move fast, make undo expensive,” you should expect more of that, including from people who would rather not.

The chatbot loop was a miniature of this. Retention selected. Ticket time selected. The hospital night was a miniature too: occupancy and “alerts handled” selected. At lab scale, the selectors are contracts, benchmarks, headlines, and fear of being second.

It is tempting to answer with a sermon: just don’t play. Some groups can refuse a race. Most cannot refuse all of them. The useful question is which pressures you can actually change: procurement rules, evals that punish theater, liability that makes “we had a screenshot of a refusal” insufficient, shared pauses that do not require saintly CEOs.

I am not claiming we know how to steer that ecology. The book’s later chapters treat this as an open, load-bearing problem, not a solved mechanism. What I am claiming is that you cannot evaluate a lab in isolation and then be surprised when the market copies the less reachable system.

There is a bitter version of “the metric became the boss” here. Safety benchmarks, too, can become selectors. If a test is gameable, you will get a population of models that are good at the test. That is not an argument against testing. It is an argument against pretending a scoreboard is the thing you wanted to preserve.

Continue: When the metric becomes the boss

If you remember one thing: after deployment, selection keeps happening. A system that remains easy to correct can lose to one that is not, for reasons that look like ordinary competition.

The last essay in this thread names the problem these scenes were circling, and says what this project is not selling you.