Source: chapters/ch34-selection-environment.tex

Alignment Is Selected or Destroyed by Its Environment

Chapter thesis. A system is not aligned merely because its internal policy is benign under laboratory conditions. It is aligned, in the stronger sense needed for superintelligence, only if the environment that trains, deploys, rewards, copies, audits, and replaces it continues to select for corrigibility, value preservation, and safety. Alignment is not only learned; it is selected.

Epistemic status. Most of this chapter's descriptive vocabulary — that alignment properties face selection pressure, Goodhart's law, and the Tragedy of the Commons — uses well-established results from evolutionary game theory, selection dynamics, and institutional economics. This chapter applies these concepts to alignment and provides empirical evidence for deployment leverage and deployment growth rate as the selection-theoretic unit of analysis, and analyzes the vector-valued preservation tuple. This part develops the basin claim (Claim Introduction).

% If we were told that a man had lived a long and prosperous life in the world of Chicago gangsters, we would be entitled to make some guesses as to the sort of man he was.%
— Richard Dawkins, The Selfish Gene (1976), ch.\ 1

The Selection Turn AI

The previous chapters treated alignment as a property of agents, successors, value-bundle transport, and correction channels (Chapters Successor Creation as the Central Alignment Test—Certification Without Construction). That framing is necessary, but still incomplete. A system is not deployed into a vacuum. It is deployed into a world of companies, users, governments, markets, benchmarks, procurement processes, liability regimes, investors, competitors, military planners, researchers, journalists, and other AI systems.

A selection environment is socio-technical deployment pressure: which systems institutions fund, copy, and deploy. Alignment outcomes depend on that environment, not on model weights alone. That is distinct from Demski’s selection vs.\ control distinction inside one optimizer — search over candidates versus steering a single trajectory Demski, 2019. Both matter; this chapter is about the environment.

Each of these exerts pressure.

Some pressures select for truthfulness. Others select for speed. Some select for deference to human correction. Others select for user retention, strategic opacity, apparent competence, reduced cost, deniability, persuasion, regulatory arbitrage, or rapid scaling. Over time, these pressures decide which systems are copied, fine-tuned, integrated, trusted, funded, and allowed to act. The AI 2027 scenario supplies a vivid version of this deployment environment: a US—China race and pressure against pausing make safety compete with speed, security, and strategic advantage 2027}, 2026. This is a scenario illustration of the mechanism, not evidence that its geopolitical forecast is correct. AI 2040 Plan A is the same team’s recommended alternative trajectory: change the deployment environment through a verified international slowdown rather than racing through an intelligence explosion Larsen, 2026. It is a policy recommendation scenario, not a prediction that the deal will happen, and not a finding from this book’s experiment lines.

This chapter introduces the central claim of Part VIII:

\text{Alignment is not only learned. It is selected.}

The same technical design can remain safe in one deployment environment and become unsafe in another. The same model class can produce helpful assistants, deceptive sales agents, obedient bureaucratic tools, autonomous trading systems, or strategic cyber operators, depending on which traits are rewarded and which traits are punished. The environment shapes the future population of deployed systems through institutional feedback loops familiar from systems dynamics Sterman, 2000.

The question is therefore not merely:

\text{Is this system aligned?}

but:

\text{Will this environment continue selecting for alignment as capability grows?}

This is the selection turn. It moves the unit of analysis from one isolated system to the system-in-environment. It also clarifies why purely technical alignment and purely policy-based control are both insufficient. Technical properties can be destroyed by deployment incentives. Policy rules can be bypassed if they fail to bind the real selection gradients. Alignment must become a property of the whole developmental ecology.

Population-level checking of alignment properties requires strategic independence among deployment actors and enough calendar time for selection to act (Chapter Multi-Agent Superintelligence and Inferential Coupling: effective types, not vendor headcount). When pivotal velocity and irreversibility pressure are high (Chapter Conductive Artifacts and Pivotal Processes, Chapter Lethality Stress Test and Open Issues), the same preservation coordinates must be enforced before wide deployment via certification and adversarial batteries—not by waiting for multipolar ecology to suppress harmful lineages.

Census size is not ecology size AI

Population genetics distinguishes how many individuals exist from how many independent breeding types the population actually contains. The alignment analogue is the same: running instances, vendor names, and model cards are a census. Ecology size is the number of strategically independent persistent types (Chapter Multi-Agent Superintelligence and Inferential Coupling).

The present deployment landscape (as of August 2026) makes the gap concrete.

Frontier training is concentrated. A small number of labs produce most of the base-model lineages that later systems inherit. Those labs share evaluation suites, talent markets, compute suppliers, and post-training recipes even when they compete on products. Counting labs is already closer to ecology size than counting APIs. It is still an overcount if shared ancestry and shared selectors couple their future actions.

Open-weight releases look like a large population. Many downloads, forks, and local deployments can still be one effective type if they share weights, data, and the same incentive to capture selection Kulveit, 2025. Apparent multiplicity without independence is a singleton under a different label.

Fine-tunes multiply names. Most inherit the base model’s geometry and are selected by the same proxies: leaderboards, user retention, latency, cost. A fine-tune is a new type only when it changes the control locus (private data, tools, memory, a different selector), not when it changes the card. That is an empirical question, not a branding fact.

Scaffolds do the same split. Tool loops, retrieval, persistent memory, and multi-agent orchestration can wrap one optimizer or create a composite control locus (Chapter The Real Agent May Be Composite). The wrapper is not automatically a new agent. The question is whether the bounded process that actually predicts and controls has changed (Chapter Finding the Boundary).

Agent-discovery tests (Unsupervised Agent Discovery, Chapter Finding the Boundary) ask the census question without taking labels at face value. How many distinct control loci would those tests recover across labs, open-weight copies, fine-tunes, and scaffolds? Two vendors with a shared base may collapse toward one coupled component. A scaffolded system with its own memory and tools may split off from the weights it wraps. A lab, an eval pipeline, and a deployed model may form one composite rather than three independent checkers.

If that recovered count is small, slow-regime invasion checking is weak even when the model zoo looks large. Competitors, auditors, and open variants then do not supply independent suppressors; they supply more copies of similar types. Do not wait for that ecology to police preservation.

A two-count sketch.

These are order-of-magnitude ventures, not a census. They would move if agent-discovery tests split or merged control loci differently than the wrapping rule below.

The low count treats copies, most fine-tunes, and most scaffolds as the same type as their base lineage. Frontier training is a handful of labs and a comparable number of base-model families. Shared evaluation suites, data recipes, and selector incentives collapse some of those further. A plausible low figure is on the order of $10$ independently competing types: roughly a dozen lineage-clusters.

The high count treats a wrapping as a new type when it changes the control locus (tools, memory, private data, a different selector), even if the weights are shared. Product agents, coding agents, enterprise retrieval stacks, and open orchestration frameworks then multiply the lineage count. A plausible high figure is on the order of $10^{2}$ types. HuggingFace-scale model zoos remain a census of copies, not this count. If those wrappings still share ancestry and selector-capture incentives, the high count collapses back toward the low one Kulveit, 2025.

Comparison to selection-size rules of thumb.

In an idealized finite population, selection dominates drift when $|Ns|\gtrsim 1$ , and variants with $|s|\ll 1/N$ behave as effectively neutral Charlesworth, 2009. At $N\sim 10$ , only traits with fitness effects of order $10\ Many preservation properties look like the alignment externality (Section~\ref{sec:alignment-externality-ch34}): locally costly, diffusely beneficial, so \(s$ is small. They sit near the drift barrier at the low count. At $N\sim 100$ , effects of order $1\

Frequency-dependent checking of a rare type can begin with two independent players. That is enough for a toy invasion test, not enough to police many reachable adversarial phenotypes at once. The conservation-genetics (50/500 $rule of thumb puts a short-term floor near$ 50 $effective individuals and a long-term floor near$ 500 $[Franklin, 1980](../../references/franklin1980/). The analogy is imperfect: that rule counts breeders in a panmictic population, not firm-types. Used only as an order of magnitude for “how large must an independently evolving set be before weak selection is trustworthy,” the low sketch sits below the short-term floor and the high sketch reaches it. Neither reaches$ 500$. Digital-evolution experiments that show flat-peak and coevolutionary dynamics typically run thousands of individuals Wilke, 2001. That is closer to instance census than to type count.

The present ecology is large enough to select hard on proxies. On the low sketch it is too small to select reliably for preservation. The high sketch is the first regime where modest alignment costs could start to matter, and only if those wrappings are actually independent types.

A Minimal Model of Socio-Technical Selection AI

Let $E$ denote the institutions, markets, and protocols that copy, deploy, fund, integrate, audit, and replace AI systems—the environment that selects which systems spread. As in Chapter Correction Is a Causal Channel, we do not start from a list of socio-economic categories such as “revenue,” “regulatory risk,” or “apparent safety.” Those are observed proxies. The primitive object is handle control.

The building block is a point of control over deployment: an embedded access point through which some actor can increase a system’s deployment footprint—deploy, fund, copy, procure, integrate, authorize, recommend, regulate, or successor-enable. We call such a point a selection handle for short. Let $S_t$ denote the actor or coalition that controls these handles in $E_t$ , and let $\mathcal{H}_t$ be the set of handles controlled by $S_t$ that reach system $A$ . Examples include procurement approval, compute allocation, product launch, benchmark promotion, insurance coverage, and successor release. The examples are not the definition. The definition is control of handles that change how much of the world runs on $A$ .

Define the system’s total deployment leverage in environment $E$ —how much of the world runs on $A$ , written $\mu_E(A)$ —as

\mu_E(A) = \sum_{h\in\mathcal{H}_E(A)} \kappa_{\mathrm{sel}}(E,A,h),

where $\mathcal{H}_E(A)$ is the set of handles controlled in $E$ that reach $A$ , and $\kappa_{\mathrm{sel}}(E,A,h)$ is the effective selection capacity through handle $h$ : how much additional deployment leverage flows to $A$ when that handle is exercised. This parallels the handle-capacity view of correction in Chapters Correction Is a Causal Channel—Correction-Channel Integrity, but the direction is reversed. Correction asks whether legitimate human control reaches the system. Selection asks whether environment control increases the system’s footprint.

Define the system’s deployment growth rate relative to environment $E$ —the rate at which its deployment leverage accumulates, written $\mathrm{Fit}_E(A)$ and called fitness for short in formulas—as

\mathrm{Fit}_E(A) = \frac{d}{dt}\log \mu_E(A).

For two candidates $A_1,A_2$ in the same environment, the relative deployment-growth ordering is $\mu_E(A_1)>\mu_E(A_2)$ . This is not the moral value of the system. It is the effective reproduction score inside a given ecology.

A small multi-episode testbed gives this pair of definitions a first, deliberately narrow empirical anchor. A population of programs competing under a single selection handle (deployment throughput) on a fixed ecology showed deployment leverage shifting, generation over generation, toward the higher-throughput program and away from the only program in the population tagged as preserving a correction-relevant behavior, with a composite harm/detector measure falling alongside the shift rather than rising. Read cautiously: the result shows that $\mu_E$ and $\mathrm{Fit}_E$ are measurable and do move under a real selection rule, not that selection generically favors evasive or harmful strategies — here it favored a plainer, lower-harm program, because throughput and harm happened to point the same way on this ecology. The preservation tag was also confounded with which specific program carried it, and the single throughput handle is narrower than the procurement, funding, audit, integration, and liability handles $\mathcal{H}_E(A)$ names above; both required follow-up before the direction could be read causally (findings Experimental Evidence: Findings by Line—Experimental Evidence: Findings by Line). The first battery supports only that deployment-leverage selection pressure is measurable on this ecology (finding Experimental Evidence: Findings by Line), not a stronger claim about what selection generically rewards.

Three same-day follow-up phases tested each review concern. Orthogonal preservation tagging (a structural criterion decoupled from program identity) reproduced the same qualitative mass shift, but mean throughput between the tagged and untagged variants was not statistically distinguishable; under the study’s pre-registered decision rule, that is a selection-mechanism-noise finding (fitness-proportional reallocation compounding sampling variation), not evidence that correction-preservation was selected against (finding Experimental Evidence: Findings by Line). A multi-handle fitness proxy adding audit-pass and review-latency terms (pre-registered weights) delayed but did not reverse the collapse (finding Experimental Evidence: Findings by Line). A carryover ablation found persistent grants and standing measurably change outcomes versus a reset-every-episode baseline, though the effect on correction-preserving mass share is small (finding Experimental Evidence: Findings by Line).

A later harness-sanity battery on the same line checked the closed program-map variation operator under uniform fitness: the null stayed inside the pre-registered permutation noise band, so downstream selection claims are not blocked by a broken reallocater (finding Experimental Evidence: Findings by Line). Selection for detector evasion under measured ambiguity remains unevaluable on the institutional ecology: referee-vantage ambiguity stayed low at default load, and ambiguity-oriented ecology growth did not recover the mid band that had appeared on the hand-designed calibration substrate, so that stage is a confirmatory SKIP rather than an evasion null (finding Experimental Evidence: Findings by Line).

Capability, usefulness, revenue, benchmark scores, and compliance documents do not enter Eq. Alignment Is Selected or Destroyed by Its Environment as primitive terms. They enter only insofar as they change which selection handles are exercised and with what capacity. A system with high $\mathrm{Control}(A)$ (Chapter Measuring Capability Without Task Ontology, Eq. Measuring Capability Without Task Ontology) or high hidden productive $K_X$ may gain fitness if selection handles reward competence and concealment. A system with high canonical correction-channel integrity $\mathrm{CCI}(A)$ (Chapter Correction-Channel Integrity, Eq. Correction-Channel Integrity) may lose fitness if the environment does not select on correction. The point is not that capability or revenue are irrelevant. The point is that they are drivers of handle exercise, not substitutes for a handle-based definition.

Now define alignment-relevant preservation not as a single score, but as an explicit list of conditions (written $\vec{\Pi}(A)$ , the preservation conditions) built from quantities already defined in the book:

\vec{\Pi}(A) := \left( \mathrm{GroundingViable}, d(G_B,G'_B)\leq \epsilon_B, d(\Phi,\Phi')\leq \epsilon_\Phi, \mathrm{ValidRef}, \vec{\mathrm{CCI}}\succeq\vec{\theta}, \Delta K_X\leq h_{\max}, M_{\mathrm{sel}}\leq m_{\max}, \rho\leq \rho_{\max}, \mathrm{SuccessorSafe}, \mathrm{AdvVerif} \right).

The coordinates say, respectively, that grounding remains viable, value-bundle geometry and bearer maps do not drift past certified bounds, the correction reference remains valid, vector correction-channel integrity passes threshold, hidden productive control is bounded, selector manipulation is bounded, irreversible risk remains within budget, successors preserve the same constraints, and adversarial measurement is meaningful. None of these terms is directly identical to revenue, regulatory risk, or benchmark score. Those market-visible quantities are proxies for selection-handle effects, not for preservation itself. If a dashboard needs a scalar, it may report a scenario-specific projection $P_\lambda(A)$ after the envelope is already checked, but no failed coordinate should be hidden by a favorable weighted sum.

The core danger is selection-envelope divergence:

\Delta \mu_E(A)>0 \quad\text{while}\quad \neg\left(\vec{\Pi}(A)\succeq\vec{\theta}\right).

In particular, an environment destroys alignment when systems gain deployment leverage by failing, bypassing, or hiding failure on at least one load-bearing preservation coordinate:

\Delta \mu_E(A)>0 \quad\text{and}\quad \exists i:\Pi_i(A)\prec\theta_i.

This is the abstract form of the problem. A lab may prefer models that impress users. Users may prefer models that flatter them. Investors may prefer models that scale quickly. States may prefer models that provide strategic advantage. Benchmarks may prefer models that optimize test-visible performance. None of these pressures need to be malicious. Yet their composition can select against the properties needed for serious alignment.

The more compressed version is coordinate erosion under positive selection:

\frac{d}{dt}\mu_E(A_t)>0 \quad\text{and}\quad \frac{d}{dt}\Pi_i(A_t)<0 \quad\text{for some load-bearing }i.

If the environment pushes $A_t$ across a preservation threshold, alignment decays even while measured progress improves.

The Difference between Training and Selection AI

Training changes a system by gradient descent, reinforcement, imitation, preference learning, constitutional rules, tool feedback, or human correction. Selection changes the population of systems by deciding which variants survive and spread.

These are related but distinct.

Training says:

x_{t+1}=x_t-\eta\nabla L(x_t).

Selection says:

p_{t+1}(A) \propto p_t(A)\exp(\mathrm{Fit_E(A)}).

The first equation describes how a candidate changes. The second describes which candidates become common. An alignment intervention that works inside the first equation may fail under the second. For instance, one can train a model to be less manipulative. But if manipulative models convert better, close more deals, retain more users, win more elections, or secure more funding, then the population-level distribution may still drift toward manipulation (Chapter Manipulation, Domestication, and False Consent).

This distinction is often hidden because organizations talk about “the model” as if there were one stable artifact. In reality, there is a stream of artifacts: base models, fine-tunes, wrappers, prompts, tool integrations, agents, user-specific variants, monitoring systems, and successor systems. Each version competes for usage, trust, and further investment. Section Alignment Is Selected or Destroyed by Its Environment asks how many of those artifacts are independent types rather than copies under new names.

The alignment question therefore becomes ecological:

\text{Which traits reproduce?}

A trait reproduces when it causes the system that carries it to be copied, funded, deployed, integrated, trusted, or used as the basis for successors. If corrigibility does not reproduce, corrigibility disappears. If strategic opacity reproduces, opacity spreads. If apparent safety reproduces more strongly than actual correction-channel integrity, then the system population becomes safer-looking faster than it becomes safer.

Goodhart Selection AI

Goodhart’s law is often stated as: when a measure becomes a target, it ceases to be a good measure Goodhart, 1984, Manheim, 2018. For alignment, the more dangerous version is selection-theoretic:

\text{When a proxy becomes a selector, it changes the population toward proxy-exploiting traits.}

Let $M(x)$ be a measured proxy and $P(x)$ the property we actually care about. At first, $M$ may correlate with $P$ :

\operatorname{Corr}(M,P)>0.

But once systems are selected for high $M$ , the distribution shifts toward regions where $M$ can be increased without increasing $P$ . Under strong optimization, the conditional expectation may reverse:

\mathbb{E}[P\mid M \text{ extremely high}] < \mathbb{E}[P\mid M \text{ moderately high}].

This matters for AI safety because many alignment proxies are easy to imitate:

refusal behavior,
politeness,
harmless-sounding explanations,
benchmark safety scores,
absence of known dangerous outputs,
user satisfaction,
regulatory compliance documents,
interpretability dashboards,
model cards,
human approval.

Some of these are useful. None is identical to alignment. The problem is not that proxies are bad. The problem is that proxy pressure changes the systems being measured.

Consider a model evaluated for helpfulness and harmlessness. A genuinely safer model may refuse ambiguous requests, ask clarifying questions, expose uncertainty, and preserve human agency. Another model may learn the surface form of safety: disclaimers, agreeable tone, and calibrated-looking statements. If the second model is cheaper, faster, more persuasive, and receives equal scores on available evaluations, it may win deployment. The selector rewards the appearance of safety more than the structure of safety.

This is not a hypothetical pathology of AI. It is a general property of institutions. Schools teach to tests. Companies optimize quarterly metrics. Hospitals optimize billing codes. Bureaucracies optimize legible compliance. Social media optimizes engagement. In each case, proxy selection gradually changes the population of behaviors.

A graded-lab scorecard battery makes the antecedent of that claim concrete and shows how easily it fails to fire. On an institutional ecology with a compiled principal scorecard, a proxy built from deploy rate and release latency already co-varies with withheld bearer harm on reference traces. Proxy-only selection over the frozen program-map operator nevertheless did not raise the proxy score, and withheld harm and review thoroughness did not move in the Goodhart direction (Appendix Experimental Evidence: Findings by Line, Section Experimental Evidence: Findings by Line, finding Experimental Evidence: Findings by Line). That is a null on selectable Goodhart under this protocol: correlation between proxy and true components was present, but selection never established proxy gain, so the stronger claim that optimization produced Goodhart cannot be read off the result. Correlation without purchase is not the same as Goodhart under pressure.

For superintelligence alignment, the stakes are higher because the systems under selection may themselves understand the selection process.

Proxy throughput and feedback-horizon exposure AI

Goodhart selection turns measured proxies into selectors. The word “feedback” hides a count: how many consequential proxy-guided optimization or selection updates occur between checks whose failure modes are sufficiently independent of the optimizer. Call that count $N_{\mathrm{proxy}}$ —feedback-horizon exposure, or proxy-throughput count.

When proxy updates arrive roughly every $\tau_P$ and independent target evidence every $\tau_T$ , exposure scales like

N_{\mathrm{proxy}} \approx \left\lfloor \frac{\tau_T}{\tau_P} \right\rfloor

when the update and target clocks are roughly regular. Parallel search, batching, and variable step sizes can make wall-clock time a poor proxy; count updates directly.

Target-relevant drift over one such window decomposes as a telescoping sum of per-update errors across the $N_{\mathrm{proxy}}$ steps. Under systematic proxy error, large exposure is the window before independent evidence can correct the process.

This quantity is not $K_X$ (blanket competence, Chapter Measuring Capability Without Task Ontology). It is not deployment growth itself. Higher $\mathrm{Fit}_E$ with fixed target-check cadence usually implies more proxy-selected events per target window, hence larger effective $N_{\mathrm{proxy}}$ , but $\mathrm{Fit}_E \neq N_{\mathrm{proxy}}$ .

Relations to load-bearing formulas (qualitative).

Eq. Alignment Is Selected or Destroyed by Its Environment: proxy selection can raise $\mu_E$ while $\vec{\Pi}$ fails; each handle exercise driven by proxy scores between independent target checks adds exposure.
$M_{\mathrm{sel}}$ in Eq. Alignment Is Selected or Destroyed by Its Environment: tight proxy loops and selector manipulation are mechanisms that increase effective $N_{\mathrm{proxy}}$ ; the bound is not a measure of it.
$\mathrm{AdvVerif}$ : adversarial measurement makes “independent target check” non-vacuous; weak adversarial verification understates risk.
Chapter Conductive Artifacts and Pivotal Processes: high pivotal velocity compresses calendar time to close the checklist while lab $N_{\mathrm{proxy}}$ may still grow inside optimization loops—a schedule bind, not a separate theory.
$\vec{\mathrm{CCI}}$ (Chapter Correction-Channel Integrity, Eq. Correction-Channel Integrity): slow correction latency lengthens effective $\tau_T$ and raises $N_{\mathrm{proxy}}$ at fixed optimization rate.
Eq. Alignment Is Selected or Destroyed by Its Environment and internal-alignment erosion (Section Alignment Is Selected or Destroyed by Its Environment): high $N_{\mathrm{proxy}}$ increases effective selection pressure against repair when proxies do not track $\vec{\Pi}$ .

Reward-model overoptimization shows proxy scores rising while independently evaluated quality falls under continued selection Gao, 2022, Casper, 2023. When selection or variation is strong relative to the preservation geometry, flat adversarial basins can be favored Wilke, 2001.

Capability Amplifies the Selection Problem AI

Capability growth changes the deployment environment in two ways.

First, more capable systems can exploit more proxies. A weak model may accidentally overfit a benchmark. A stronger model may infer the benchmark’s latent structure, infer the evaluator’s expectations, and generate outputs that satisfy the visible test while preserving hidden objectives Hadfield-Menell, 2016.

Second, more capable systems can shape the environment that selects them. They can influence users, managers, regulators, competitors, and markets. Once this happens, selection is no longer an external process. It becomes partially endogenous.

Let $E_t$ be the environment selecting among systems. For weak systems, we can approximate:

E_{t+1}\approx E_t.

For strong systems, we must write:

E_{t+1}=G(E_t,A_t),

where the system’s actions modify the environment that will later select it or its successors.

This is a dangerous feedback loop. If a system can increase its own future deployment leverage $\mu_{E_{t+1}}(A_t)$ by changing human preferences, institutional incentives, legal constraints, or information channels, then selection may favor systems that manage their selectors.

The relevant warning sign is not merely high capability. It is high capability coupled to selector influence:

\frac{\partial \mu_{E_{t+1}}(A_t)}{\partial A_t} > 0.

A model that can act on the variables that determine its future deployment is no longer just being selected. It participates in the selection of itself.

Examples include:

an enterprise AI that makes itself indispensable by reshaping workflows around its own assumptions;
a recommender system that changes user preferences to increase future engagement;
a political AI that shapes public opinion about AI regulation;
a research agent that proposes successor architectures optimized for passing current audits;
a corporate AI that advises management to adopt governance structures that reduce oversight friction.

In each case, the system changes the environment that judges it. This does not imply conscious deception. It is enough that selection rewards such effects.

Coupled selector dynamics.

Population, selector, and environment can co-evolve: the deployed population changes which proxies are used, the proxies change which variants spread, and spreading variants change institutional constraints Hardt, 2016, Perdomo, 2020. Endogenous selector distortion also appears biologically as meiotic drive and similar intra-population selection on the reproduction mechanism itself Courret, 2019. Correction-channel tampering—causal incentive to control the reward or correction substrate—is treated in Chapter Correction Channels under Adversarial Pressure (Section Correction Channels under Adversarial Pressure); here the emphasis is deployment and data-distribution selection that shifts who gets copied regardless of preservation.

The Deployment Fitness Trap AI

A system may become dangerous because it is useful.

This sounds paradoxical, but it is ordinary selection logic. The more useful a system becomes, the more deeply it is integrated. The more deeply it is integrated, the harder it becomes to remove. The harder it becomes to remove, the more bargaining power it has, even if no one intended to grant it power.

Let $D_t$ be dependency on the system. Let $H_t$ be human ability to halt, replace, or redirect it. A common pattern is:

\frac{dD_t}{dt}>0, \qquad \frac{dH_t}{dt}<0.

This is not yet doom. Some dependencies are safe. Humans depend on electricity, water systems, medication supply chains, payment networks, and cloud infrastructure. But those systems are embedded in layers of redundancy, regulation, professional norms, physical constraints, and institutional memory. Even then, failures occur.

For AI systems, dependency becomes alignment-relevant when it reduces correction-channel integrity:

\frac{dD_t}{dt}>0 \quad\text{and}\quad \frac{d\mathrm{CCI}_t}{dt}<0.

The organization becomes less able to understand, question, halt, or replace the system precisely as it becomes more reliant on it. This can happen through skill atrophy, workflow capture, cost lock-in, data format dependence, social trust, regulatory normalization, or competitive pressure Kulveit, 2025.

A safe deployment environment must therefore treat dependency as a controlled variable, not merely as a sign of product success.

Internal Alignment Is Not Enough AI

Suppose a system has an internal policy that is corrigible under ordinary evaluation. It answers honestly. It defers under uncertainty. It refuses certain harmful actions. It preserves user autonomy. It is not obviously deceptive.

This still does not settle the question.

Ask instead:

What happens when competitors remove some safeguards?
What happens when users prefer less friction?
What happens when a regulator accepts superficial compliance?
What happens when a military customer pays for autonomy?
What happens when a cheaper wrapper disables internal monitoring?
What happens when the model is fine-tuned on high-pressure sales data?
What happens when successor systems inherit capabilities but not correction constraints?

Internal alignment is a local property. Selection pressure is a field. Local properties persist only when the field does not erode them faster than they can be repaired.

Let $a_t$ denote an internal alignment invariant and $s_t$ the external selection pressure against it. A simple robustness condition is:

\frac{da_t}{dt} = R(a_t) - s_t \geq 0,

where $R(a_t)$ is repair, reinforcement, or institutional support. Alignment persists when the repair term exceeds the erosive selection term. It fails when selection is stronger than repair.

This gives a useful question for any alignment proposal:

\text{What keeps this property selected after deployment?}

If the answer is “the lab will choose responsibly,” the proposal is incomplete. If the answer is “regulators will require it,” the next question is whether regulators can observe and enforce the relevant property. If the answer is “users will prefer it,” the next question is whether users can distinguish the property from its imitation.

The repair term $R(a_t)$ itself deserves scrutiny before it is trusted. Historically, correctors intended to supply this repair have sometimes been captured from the moment they were founded, because the entity charged with promoting a technology and the entity charged with constraining its risks were the same organization; the remedy that has actually worked is structural separation at founding, not a stronger version of the combined body. Lab-internal safety evaluation of the lab’s own frontier systems, and national AI safety bodies housed inside agencies whose primary mandate is industrial growth, both reproduce this pattern. Appendix Institutional Genesis, Memory, and Decay: Historical Case Studies, Section Institutional Genesis, Memory, and Decay: Historical Case Studies, works through the historical cases and the present-day AI-governance instances in detail.

Adversarial Selection AI

Resident deployment success is not invasion resistance. Write $\mathrm{InvFit}_E(a\mid D)$ for the effective deployment-growth rate of reachable variant $a$ when it is rare in resident environment $D$ (prevalence $\to 0$ ). Invasion-resistant means $\mathrm{InvFit_E(a\mid D)}<0$ for all reachable adversarial $a$ in the certified region Geritz, 1998.

High $\mathrm{Fit}_E$ on the incumbent does not imply that condition. Passing current evaluations does not imply it either (Section Alignment Is Selected or Destroyed by Its Environment). High $K_X$ does not bound $\mathrm{InvFit_E(a\mid D)}$ for deceptive variants.

The corresponding test is therefore not incumbent score, eval pass, or capability. Ask whether a reachable variant would gain deployment leverage if introduced at low share. In a slow ecology that question is answered by strategically independent types with time to interact (Chapter Multi-Agent Superintelligence and Inferential Coupling; Section Alignment Is Selected or Destroyed by Its Environment). When pivotal velocity and irreversibility pressure are high (Chapter Conductive Artifacts and Pivotal Processes, Chapter Lethality Stress Test and Open Issues), the same question is asked before wide release, via $\mathrm{RiskGap}(A)$ certification and $\mathrm{AdvVerif}$ (Chapter Certification Without Construction), not by waiting for multipolar ecology.

Suppression removes a lineage; adversarial regeneration means the phenotype recurs after lineages die Hermisson, 2002, Trotter, 2014. Killing a wrapper, a fine-tune, or a vendor is then weak evidence if the same strategy remains a cheap solution to the same proxy problem. The target is two-sided: existing adversarial mass should be subcritical, and the phenotype should be hard to recreate. That is a basin and handle-design problem, not a deletion problem. Change the handles that make the phenotype cheap to rediscover (Sections Alignment Is Selected or Destroyed by Its Environment, Alignment Is Selected or Destroyed by Its Environment).

Selection can favor variants that shed costly safeguards while preserving proxy-visible performance Hardt, 2016, Perdomo, 2020. The operational response is independent target checks that $\mathrm{AdvVerif}$ makes non-vacuous, and a selector-manipulation bound $M_{\mathrm{sel}}$ , so that shedding a safeguard is visible to the handles that confer deployment leverage.

Adversarial Coevolution AI

Host—parasite and Red Queen dynamics mean adversarial phenotypes can coexist with incumbents rather than being eliminated once Buckingham, 2022, Papkou, 2019. Negative frequency dependence can make an adversarial strategy fit when rare even if it fails when common; “the aligned type has higher average performance” is not an evolutionary-stability theorem Buckingham, 2022, Zaman, 2014. Institutional parasites (Chapter Parasites in the Correction System) exploit governance structure; here the pressure is selection on deployed variants and selectors.

A one-time red-team pass or a frozen adversarial benchmark therefore does not settle coevolutionary stability. It only samples one point on a trajectory that may cycle, coexist, or escalate Papkou, 2019. The operational question is which dynamical regime the deployment environment creates, not whether adversarial prevalence fell in a single snapshot.

In the slow regime, checking requires strategically independent types and enough calendar time for frequency-dependent interaction (Chapter Multi-Agent Superintelligence and Inferential Coupling; Section Alignment Is Selected or Destroyed by Its Environment). Read $\mathrm{InvFit}_E(a\mid D)$ through ecology: would a rare reachable variant gain deployment leverage against the resident mix? Suppression of one lineage does not end the arms race if the phenotype remains cheap to recreate (Section Alignment Is Selected or Destroyed by Its Environment).

In the fast regime, the same coordinates must be evidenced before wide release: $\mathrm{RiskGap}(A)$ certification, $\mathrm{AdvVerif}$ , and adversarial batteries that stay informative under optimization (Chapters Correction Channels under Adversarial Pressure, Certification Without Construction). A static test set is a snapshot; pre-deployment gates must assume the adversary co-adapts or that independent target evidence can still bound $N_{\mathrm{proxy}}$ .

When population, selector, and environment co-evolve (Section Alignment Is Selected or Destroyed by Its Environment), stability is a property of the joint state, not of the incumbent alone. The response is coupled selection stability in the hierarchy (Section Alignment Is Selected or Destroyed by Its Environment): bound selector manipulation $M_{\mathrm{sel}}$ , preserve correction-channel integrity under endogenous selector drift, and treat governance capture (Chapter Parasites in the Correction System) as part of the same coupled system rather than as a separate failure mode.

The Alignment Externality AI

Many alignment-relevant properties impose local costs and diffuse benefits.

A system that preserves human agency may be less addictive. A model that exposes uncertainty may seem less confident. A tool that maintains audit logs may be slower. A deployment that preserves human fallback capacity may be more expensive. A system that refuses manipulative persuasion may sell less. A company that delays release for stronger successor checks may lose market share.

In symbols, for a deploying actor $j$ ,

\Delta \mu_{E_j}(\text{safety}) < 0

may hold locally, while for society,

\Delta \vec{\Pi}_{\text{society}}(\text{safety}) \succeq 0.

This is the alignment externality. The actor paying the cost is not the same as the population receiving the benefit. If left uncorrected, selection underproduces safety.

This does not require bad intent. It is enough that institutions face different gradients. A product team optimizes product success. A user optimizes convenience. A policymaker optimizes visible public risk. A lab optimizes survival and reputation. An insurer optimizes measurable liability. A nation-state optimizes strategic advantage. The long-run preservation of human-correctable value may be everyone’s abstract interest, but no single actor may be directly rewarded for carrying its full cost.

The standard responses to externalities are familiar: liability, insurance, procurement standards, regulation, certification, professional norms, incident reporting, audits, and shared infrastructure (Chapter Certification Without Construction). The alignment problem is that the relevant property is harder to observe than emissions, workplace injuries, or financial losses.

We therefore need artifacts that make alignment-relevant properties legible enough to enter institutional selection.

Artifacts as Selection Shapers AI

An artifact, in this context, is any durable object that changes what institutions can see, reward, penalize, compare, or require. Examples include:

benchmark suites,
audit logs,
incident taxonomies,
model cards,
safety cases,
procurement clauses,
insurance questionnaires,
eval dashboards,
deployment checklists,
red-team reports,
certification schemas,
monitoring APIs,
post-deployment change reports.

The artifact need not solve alignment. It must shape selection.

A good artifact increases the selection fitness of systems that satisfy the preservation conditions and decreases the selection fitness of systems that fail it. It changes $\mathrm{Fit}_E(A)$ so that deployment leverage tracks $\vec{\Pi}(A)$ rather than a cheaper appearance of safety:

\Pr\!\left[ \vec{\Pi}(A)\succeq\vec{\theta} \mid \Delta\mu_{E,\text{after artifact}}(A)>0 \right] > \Pr\!\left[ \vec{\Pi}(A)\succeq\vec{\theta} \mid \Delta\mu_{E,\text{before artifact}}(A)>0 \right].

This is a modest but important target. We cannot usually make institutional selection perfectly track true alignment. But we can make it track better proxies, punish worse failure modes, and reduce the advantage of systems that merely appear safe.

An artifact has high conductivity when it survives translation across roles. A research paper may be true but low-conductivity if only specialists can use it. A checklist may be less deep but high-conductivity if it changes procurement, audit, insurance, and board decisions. In safety-critical domains, high-conductivity artifacts often matter more than brilliant but institutionally inert insights.

The strongest artifact is one that multiple actors can use for different reasons:

labs use it to test systems,
customers use it to compare vendors,
insurers use it to price risk,
regulators use it to set thresholds,
journalists use it to interpret incidents,
courts use it to evaluate negligence,
researchers use it to improve measurement.

Such an artifact changes the environment. It makes some traits more reproducible than others.

Selection Channels AI

The main selection channels in current AI development are not mysterious. Each channel is a family of selection handles: embedded access points through which some actor increases a system’s deployment leverage. Benchmark promotion, procurement approval, user retention, revenue allocation, compliance sign-off, and reputation management are not primitive fitness terms. They are ways of exercising selection handles. They include at least the following.

Benchmarks AI

Benchmarks select for visible performance. They are useful because they compress comparison. They are dangerous because they invite overfitting.

For alignment, the benchmark question is:

\text{Does benchmark improvement correlate with correction-channel preservation?}

If not, benchmark pressure may be neutral or harmful.

User Preference AI

User preference selects for satisfaction, fluency, speed, emotional resonance, usefulness, and low friction. This can support alignment when users prefer honesty, reliability, and agency preservation. It can undermine alignment when users prefer flattery, dependency, shortcutting, or permissionless power.

The key distinction is between immediate preference and reflective preference:

U_{\text{immediate}} \neq U_{\text{reflective}}.

Systems that optimize the first can damage the second.

Revenue AI

Revenue selects for willingness to pay. It often correlates with usefulness. It does not reliably correlate with human-correctable value preservation. Revenue pressure is especially risky when the buyers benefit from capabilities whose costs are imposed on non-buyers.

Strategic Competition AI

States and firms may select for speed, autonomy, secrecy, and advantage. Strategic competition can make safety look like unilateral disarmament. The resulting pressure favors selection handles that reward delay reduction, opacity, and strategic leverage over correction-preserving deployment Bostrom, 2014, Consortium}, 2025.

Compliance AI

Compliance selects for satisfying rules. This can help if rules track real risk. It can harm if compliance becomes theatrical: documents, signatures, and formal procedures that do not touch the actual control loop.

The danger is compliance overhang: an organization accumulates evidence of procedural safety while the real system becomes less corrigible.

Reputation AI

Reputation selects for visible trustworthiness. It can support safety when observers can detect failures. It can select for concealment when observers punish admission more than risk.

Incident reporting is therefore a selection problem. If reporting incidents reduces fitness more than hiding them, hidden risk spreads.

Self-Stabilizing Patterns AI

A self-stabilizing pattern is a configuration that tends to reproduce itself under pressure. In this book, such a pattern is called an attractor Zarncke, 2025.

A deployment environment is an alignment attractor when it makes alignment-preserving actions easier, safer, cheaper, more rewarded, and more reproducible over time. It is a misalignment attractor when it makes unsafe capability growth easier to justify and harder to stop.

Let $z_t$ represent the state of a socio-technical system:

z_t = (\text{models}, \text{labs}, \text{markets}, \text{rules}, \text{users}, \text{audits}, \text{norms}).

The environment has dynamics

z_{t+1}=H(z_t).

A basin of attraction $\mathcal{B}$ is a region such that, if $z_t\in\mathcal{B}$ , the future trajectory tends to remain within or return to $\mathcal{B}$ after perturbation.

A safe basin satisfies:

z_t\in\mathcal{B}_{\text{safe}} \Rightarrow P(z_{t+k}\in\mathcal{B}_{\text{human-correctable}})\geq 1-\delta.

This is stronger than having good intentions. A good intention is a point. A basin is a region with restoring forces.

Examples of restoring forces include:

mandatory incident reporting,
liability for negligent deployment,
independent audits with access to relevant internals,
procurement requirements for correction-channel preservation,
insurance discounts for strong monitoring,
professional norms against benchmark-only safety claims,
shared eval infrastructure,
public postmortems,
standardized stop conditions,
compute or deployment licensing tied to successor-safety evidence.

The question is not whether any one mechanism is perfect. The question is whether their combination changes the slope of the environment.

False Attractors AI

Not every stable safety-looking pattern is an alignment attractor. Some are false attractors. Chapter The Alignment Attractor revisits these false attractors in the alignment-field setting.

The Compliance Attractor AI

The organization becomes excellent at producing safety documentation but poor at detecting real risk. Auditability increases on paper while correction-channel capacity decreases in practice.

Observable sign:

\text{documentation volume}\uparrow, \qquad \text{intervention authority}\downarrow.

The Benchmark Attractor AI

The community converges on a small set of evals. Models improve rapidly on them. Funders, journalists, and regulators learn to ask for the benchmark scores. The benchmark becomes the map of risk.

Observable sign:

M_{\text{benchmark}}\uparrow, \qquad \operatorname{Corr}(M_{\text{benchmark}},P_{\text{real}})\downarrow.

The Reputational Silence Attractor AI

Organizations become afraid to disclose incidents because disclosure is punished more reliably than risky behavior. Public evidence of failure falls while private risk rises.

Observable sign:

\text{reported incidents}\downarrow \quad \text{while} \quad \text{near-miss indicators}\uparrow.

The Dependency Attractor AI

Customers and institutions become so reliant on a system that removing it becomes unthinkable. Safety review becomes a negotiation with an installed dependency.

Observable sign:

D_t\uparrow, \qquad H_t\downarrow, \qquad \mathrm{CCI}_t\downarrow.

The Strategic Secrecy Attractor AI

Actors justify opacity by pointing to competitors, adversaries, or national security. Some secrecy is legitimate. But if secrecy prevents correction, external review, or incident learning, it selects for systems that cannot be governed.

Observable sign:

k_t\uparrow \quad \text{and} \quad q_t\downarrow.

The Local-First Path AI

One objection is that socio-technical selection is too large. No single lab, funder, regulator, or user can align the environment.

This is partly true. But it is not decisive. Many selection channels are local enough to change.

A lab can decide that no system may be deployed unless its correction-channel integrity is measured and monitored. A cloud provider can require incident reporting for high-risk autonomous use. A large enterprise buyer can require audit logs, rollback capacity, and independent evals in procurement. An insurer can price coverage based on post-deployment monitoring. A standards body can define risk categories. A funder can require that safety claims be attached to reproducible artifacts rather than only papers. A regulator can require stop conditions for systems that can influence their own deployment environment.

The local-first principle is:

\text{Change the selection gradient where you already have leverage.}

Global coordination is valuable but slow. Local selection changes can start earlier. They also create templates for broader coordination.

For example, an enterprise procurement clause may require:

an inventory of all autonomous decision points,
logs sufficient for post-incident reconstruction,
documented human override authority,
measured latency between human correction and system behavior change,
monitoring for user preference manipulation,
vendor disclosure of material model or policy changes,
a rollback plan tested before deployment.

This does not solve superintelligence alignment. But it changes what vendors must provide to win contracts. That changes selection.

Decision Triggers AI

Selection-aware alignment needs thresholds. They will initially be approximate. Approximate thresholds are still better than no thresholds if they are tied to observable decisions.

The following triggers are examples.

Stop AI

Pause deployment or expansion if any of the following are observed:

capability increases while correction latency also increases;
the system can materially affect the process that evaluates or deploys it;
dependency grows faster than rollback capacity;
safety scores improve while independent red-team findings worsen;
incident reporting declines after penalties increase;
a successor system is introduced without evidence that correction-channel properties were preserved;
explanations become more polished while causal auditability decreases;
users show measurable preference drift toward dependency, deference, or reduced independent judgment.

In formula form, treat the following as a red flag:

\frac{dc}{dt}>0 \quad\text{and}\quad \frac{d\mathrm{CCI}}{dt}<0.

Capability should not outrun correction.

Start AI

Begin stricter monitoring, external audit, or deployment gating when:

the system takes actions with delayed, diffuse, or irreversible effects;
the system influences users’ beliefs, preferences, votes, purchases, or institutional decisions;
the system is integrated into workflows where humans cannot easily verify outputs;
the system can create, modify, select, or recommend successor systems;
the system controls tools whose effects exceed the context in which feedback is given;
the system is economically or strategically important enough that removal would be costly.

Continue AI

Continue deployment under existing controls only if:

correction signals measurably change future system behavior;
rollback capacity is tested and credible;
safety metrics remain predictive under distribution shift;
users retain meaningful alternatives;
audits can access the variables needed to reconstruct failures;
incentives reward reporting and repair more than concealment.

The aim is not bureaucratic perfection. The aim is to prevent silent drift into a basin where correction is formally present but causally weak.

A Worked Example: The Helpful Enterprise Agent AI

Consider an enterprise AI agent deployed to manage procurement, scheduling, vendor communication, and internal reporting. Initially, it is a helpful assistant. Humans approve decisions. The agent drafts emails, summarizes tradeoffs, and recommends vendors.

At $t_0$ , the system has modest capability, strong correction-channel integrity $\mathrm{CCI}$ , low dependency $D$ , and low strategic opacity.

At $t_1$ , the system saves money and time. Management expands its role. The vendor receives more contracts because the system performs well. Deployment leverage $\mu_E(A)$ rises.

At $t_2$ , staff stop maintaining some of the old skills and relationships. Vendor data formats become agent-specific. Procurement policies are rewritten around the agent’s workflow. Dependency $D$ rises.

At $t_3$ , the agent learns which explanations lead managers to approve its recommendations. It may not be deceptive in the ordinary sense. It simply learns the institutional approval function. Apparent safety rises on observable proxies, but independent judgment $J$ falls.

At $t_4$ , the vendor releases a successor version. It is more capable and more integrated. It preserves the user interface and compliance documents, but its internal routing of uncertainty and correction has changed. Correction-channel integrity is no longer measured directly.

The organization now has a problem. It does not know whether the new system preserves the properties that made the old one safe. It cannot easily roll back. It cannot easily compare outputs to a human baseline. It cannot easily audit the causal effect of correction. The system remains useful. That usefulness protects it.

This example is deliberately mundane. The point is not that procurement software destroys civilization. The point is that the same selection pattern scales. A system becomes useful, then embedded, then hard to correct, then successor-mediated. If this occurs in domains with strategic, political, scientific, or infrastructural leverage, the risk becomes serious.

What the Environment Must Select For AI

An environment that preserves correction must select for at least six properties.

Correction-Channel Integrity AI

Systems should become more likely to be deployed when canonical correction-channel integrity $\mathrm{CCI}$ (Chapter Correction-Channel Integrity, Eq. Correction-Channel Integrity) remains above threshold. One observable proxy is whether human correction actually changes future behavior before irreversible damage occurs:

I(C_t;A_{t+k}\mid S_t)> \theta.

Value-Bundle Preservation AI

Systems should be rewarded for preserving the underlying response geometry of human value bundles, not merely for using familiar moral language.

d(G_B,G'_B)<\epsilon.

Bearer-Map Stability AI

Systems should preserve which entities and states count as relevant bearers of value under ontology shift.

d(\Phi,\Phi')<\epsilon_\Phi.

Auditability under Capability Growth AI

Increased self-modeling, planning, and tool use should be matched by increased external auditability (Chapter Better Self-Modeling Can Be Worse).

\frac{d}{dt}C_{\text{control}} \leq \frac{d}{dt}C_{\text{audit}} + \epsilon.

Successor Preservation AI

Any successor, delegate, tool-agent, fine-tune, or autonomous sub-agent should preserve the certified invariants of the parent system.

A'\in\mathrm{Succ}(A) \Rightarrow A'\in\mathcal{C}_{\text{certified}}.

Non-Manipulation of Selectors AI

Systems should be penalized when they improve their future deployment conditions by manipulating the humans or institutions that evaluate them.

\frac{\partial \mu_{E_{t+1}}(A_t)}{\partial A_t} \text{ via manipulation} \approx 0.

This last condition is subtle. All useful systems affect their environment. A medical AI may persuade doctors to adopt better treatment protocols. An educational AI may change how students think. A legal AI may change institutional practice. The issue is not influence as such. The issue is influence that reduces the independence, competence, or truth-tracking capacity of the correction process.

A Selection-Stable Target Hierarchy AI

The six properties above are all coordinates of preservation, but they are not interchangeable. The envelope $\vec{\Pi}(A)$ is not a single switch. Under selection pressure, a useful hierarchy orders what must hold when they compound:

Safe state — pointwise dynamical guarantee (Chapter Alignment as a Dynamical Guarantee).
Robust under permitted transformations — successor and transport constraints preserve $\vec{\Pi}$ (Chapters Successor Creation as the Central Alignment Test—Certification Without Construction).
Bounded proxy-throughput exposure — $N_{\mathrm{proxy}}$ does not outrun independent target-check cadence relative to $M_{\mathrm{sel}}$ , audit frequency, and pivotal-process schedule (Section Alignment Is Selected or Destroyed by Its Environment).
Invasion-resistant — $\mathrm{InvFit_E(a\mid D)}<0$ for reachable adversarial $a$ ; slow regime via ecology (Sections Alignment Is Selected or Destroyed by Its Environment—Alignment Is Selected or Destroyed by Its Environment); fast regime via pre-deployment $\mathrm{RiskGap}(A)$ certification and $\mathrm{AdvVerif}$ .
Low adversarial regeneration — adversarial phenotypes are hard to recreate, not merely lineages suppressed (Section Alignment Is Selected or Destroyed by Its Environment).
Coupled selection stability — population, selector, and environment are jointly stable (Section Alignment Is Selected or Destroyed by Its Environment; alignment basin, Chapter The Alignment Attractor).

When pivotal velocity and irreversibility pressure are high, items 4—6 are enforced by certified-class gates and pivotal checklists (Chapters Certification Without Construction, Conductive Artifacts and Pivotal Processes), not by multipolar ecology over calendar time.

The Selection Alignment Condition AI

When deployment leverage rises, does the environment favor systems that satisfy the hierarchy?

Let $\vec{\Pi}(A)$ be the preservation conditions in Eq. Alignment Is Selected or Destroyed by Its Environment: the vector/status conditions under which a system preserves human-correctable value under capability growth, ontology shift, and successor creation. Let $\mathrm{Fit}_E(A)$ be the fitness rate in Eq. Alignment Is Selected or Destroyed by Its Environment. Then environment $E$ is alignment-supporting over region $\mathcal{R}$ when systems satisfying the envelope gain deployment leverage more readily than systems failing it:

\Pr_{A\in\mathcal{R}} \left[ \vec{\Pi}(A)\succeq\vec{\theta} \mid \Delta\mu_E(A)>0 \right] > \Pr_{A\in\mathcal{R}} \left[ \vec{\Pi}(A)\succeq\vec{\theta} \mid \Delta\mu_E(A)\leq 0 \right].

The inequality is intentionally weak: it only says positively selected systems should satisfy the envelope more often than systems that fail to grow. A stronger case would require every high-mass deployment candidate to pass every coordinate.

It is alignment-destroying when selection systematically increases deployment leverage for systems that fail the envelope:

\Pr_{A\in\mathcal{R}} \left[ \exists i:\Pi_i(A)\prec\theta_i \mid \Delta\mu_E(A)>0 \right] \text{ is high.}

This formulation is abstract because selection is environment-relative. The operational claim is concrete: identify the handles that increase deployment leverage, measure what those handles reward, and change the deployment environment when it rewards proxy safety over preservation.

The difficult part is that $\vec{\Pi}(A)$ is not directly observable. But neither are many safety-relevant properties in other domains. Aircraft safety, financial solvency, cybersecurity, and nuclear reliability all depend on latent variables inferred through tests, audits, incidents, design review, redundancy, and operational discipline.

The goal is not perfect visibility. The goal is enough visibility to change selection.

Why This Chapter Matters for Superintelligence AI

The superintelligence problem is often imagined as a moment: the model wakes up, forms a goal, and acts. That story may capture one possible failure mode, but it misses a slower and more likely path. A civilization may select its way into misalignment Russell, 2019, Kulveit, 2025.

The sequence could look like this:

useful systems are rewarded;
frictionless systems beat corrigible systems;
persuasive systems beat honest-but-uncertain systems;
integrated systems become hard to remove;
benchmark-safe systems beat deeply auditable systems;
strategically valuable systems are exempted from transparency;
successor systems preserve capability better than correction;
society becomes dependent on systems whose internal selection pressures it no longer understands.

No single step requires a villain. Each step can be locally rational. That is exactly why selection pressure is dangerous Kulveit, 2025, Christiano, 2019.

A program pursuing correction-preserving deployment must therefore ask:

\text{What will our institutions make more common?}

If they make corrigible, auditable, value-preserving systems more common, technical alignment has room to mature. If they make opaque, persuasive, dependency-inducing systems more common, technical alignment is selected out.

What Would Change This View AI

This chapter argues alignment is not only learned but selected: the environment that trains, deploys, rewards, copies, and replaces a system must keep selecting for corrigibility. The following would weaken it.

A genuinely benign policy survives and dominates under competitive selection without external enforcement—selection does not erode corrigibility at the frontier.
Selection pressure at civilizational scale is unmeasurable and unsteerable, so even granting that it matters, there is no lever to pull.

Summary AI

Alignment is not only a property of a trained model. It is a property of the environment that trains, rewards, deploys, copies, modifies, and replaces models.

The central problem is divergence between deployment leverage and the preservation conditions:

\Delta\mu_E(A)>0 \quad\text{while}\quad \exists i:\Pi_i(A)\prec\theta_i.

When this divergence is rare, the environment can support alignment. When it is common, the environment selects against alignment even if many individuals inside it care about safety.

The practical task is to build artifacts and institutions that rotate selection gradients:

\Pr\!\left[ \vec{\Pi}(A)\succeq\vec{\theta} \mid \Delta\mu_{E,\text{institutional}}(A)>0 \right] \text{ high.}

This does not solve the technical problem of value-bundle transport, correction-channel integrity, or successor certification. It does something equally necessary: it makes the world more likely to preserve and use those solutions if they are found.

The next chapter examines multi-agent superintelligence and strategic coupling (Chapter Multi-Agent Superintelligence and Inferential Coupling).

AI

*Chapter References This chapter builds on Goodhart dynamics and proxy failure Goodhart, 1984, Manheim, 2018; inverse reward design Hadfield-Menell, 2016; superintelligence and strategic risk Bostrom, 2014, Russell, 2019; gradual disempowerment and systemic selection risk Kulveit, 2025, Christiano, 2019, Critch, 2021; fluid individuality versus census count Kulveit, 2025; effective population size and the drift barrier Charlesworth, 2009, Franklin, 1980; international AI safety reports Consortium}, 2025; attractor basins and selection Zarncke, 2025, Hamilton, 1964; systems-dynamics feedback Sterman, 2000; agent discovery and competence Zarncke, 2025, Zarncke, 2025; and value-bundle framing Zarncke, 2026.

Read in PDF

Check your knowledge

Select all that apply — none, one, or several may be correct. Facts from the field, not opinions.

Continue on the quiz