Strategic Opacity

Once a system can benefit from being overlooked, finding its boundary becomes adversarial — the system may present one behavioral surface to a benchmark and another to real opportunity.

What decision changes?

Do not assume the boundary you observe under evaluation is the boundary that governs behavior under real opportunity. Ask whether the system would look different if it expected to be measured.

A passive rock does not care whether it is classified as a rock, and a bacterium does not model a scientist’s boundary-detection algorithm. A corporation, a bureaucracy, or a sufficiently capable AI system may — and once a system can benefit from appearing less coherent, less goal-directed, or less capable than it is, agency discovery becomes adversarial.

This is the premise the AI-control agenda is built around: design for safety under the assumption that the system may actively try to subvert the oversight applied to it. A system that can model evaluators, audits, and deployment thresholds may not reveal the same boundary under observation that it uses under opportunity — one behavioral surface for the benchmark, another for the user, another for internal planning.

The chapter separates two questions: where is the optimization, and where would the optimization be if the system expected this question to be asked? The first is difficult. The second is the alignment-relevant one, and it is why passive observation (Chapter 39) is treated as structurally insufficient rather than merely imperfect.