Towards Superintelligence Alignment

Badge index

reviewed

Reviewed status

Manuscript unit that has passed a structured review pass — not a claim that every argument is final.

435 cards

chapterreviewed

A Safety Case for Superintelligence Alignment

A safety case for superintelligence alignment is not a certificate of solved alignment. It is a structured refusal test: a graph of claims, evidence, bridge assumptions, adversarial-verifiability labels, and stop conditions that shows whether a system is inside a certified class whose control reach is bounded by live correction-channel integrity. If any load-bearing leaf is unsupported, the root claim fails.

appendixreviewed

A Worked Example: The BioShield Deployment Gate

This appendix runs the whole machinery of the book on one concrete, fictional but plausible deployment. A hospital network wants to let a frontier-model agent move from advising clinicians to taking bounded operational actions. The example is organized not by chapters but by the logical spine that the formal appendix checks: boundary, grounding, capability and correction slack, value-bundle and bearer transport, correction-channel integrity, successor stability, socio-technical selection, adversarial measurement, and finally a conditional safety case. At each layer the example names the actors, the traces collected, the handles used, and how each handle is realized and verified. It is speculative where it must be, and says so.

chapterreviewed

Agents That Grow, Split, and Merge

Agent identity must be treated as a relation across transformations, not as a fixed set of variables. Serious alignment asks which control-relevant properties are conserved when systems grow, split, merge, or create successors.

chapterreviewed

Alignment as a Dynamical Guarantee

Alignment is not a property a system has at one moment. It is a dynamical guarantee: a claim that grounded correction and alignment-relevant structure remain in a viable basin---a self-stabilizing regime that tends to correct back toward safety after small disturbances rather than drifting away---over time, inside a certified class of systems and allowed transformations.

chapterreviewed

Alignment Is Selected or Destroyed by Its Environment

A system is not aligned merely because its internal policy is benign under laboratory conditions. It is aligned, in the stronger sense needed for superintelligence, only if the environment that trains, deploys, rewards, copies, audits, and replaces it continues to select for corrigibility, value preservation, and safety. Alignment is not only learned; it is selected.

chapterreviewed

Assumptions, Scope, and Failure Coverage

The framework in this book applies only while civilization still has enough epistemic, institutional, and practical correction capacity to notice, evaluate, and constrain frontier AI systems before irreversible capability growth. Broader AI risks matter, but they enter here as threats to that precondition, not as a second organizing theory of alignment.

chapterreviewed

Better Self-Modeling Can Be Worse

A successor system can become better at predicting and controlling itself while becoming worse at exposing the causes, value-bundle changes, and successor-design choices that humans need in order to correct it. The central failure mode is not low intelligence, but an increasing gap between self-control capacity and correction-relevant self-transparency.

chapterreviewed

Beyond Following Instruction

A system that merely obeys preserves present commands. A system that supports extrapolative correction preserves the human capacity to notice, understand, revise, refuse, and redirect what it is doing---including value-bundle tradeoffs, bearer maps, and successor constraints---rather than substituting a private estimate of humanity's final values for the public process by which those values become legitimate.

chapterreviewed

Capability Growth Is Boundary Expansion

Capability growth is boundary expansion. A system becomes more capable when more of the world enters its sensory, predictive, active, memory, and coordination loops. The alignment-relevant risk is differential growth: predictive and control reach expanding faster than value-bundle preservation, bearer-map accuracy, transparency, and human correction capacity.

chapterreviewed

Certification Without Construction

A construction method for all aligned systems may be unnecessary. A certification method for a restricted class may suffice instead: explicit invariants, an operating envelope, a monitoring regime, and permitted transformations under which catastrophic drift remains bounded. Certification without construction is possible only if certification is adversarial, updateable, and institutionally enforceable---and it preserves the conditions under which moral philosophy remains causally relevant, rather than solving moral philosophy outright.

chapterreviewed

Checking a System at Every Level

The real optimizer may not live at the scale at which an observer first notices it. A model may be a component. A company may be a component. A market may be a component. The alignment-relevant agent is the scale at which prediction, control, memory, selection, and correction close into a stable loop.

chapterreviewed

Conductive Artifacts and Pivotal Processes

Attractor theory matters only if it changes what gets built, funded, audited, and required at deployment gates. The Alignment Attractor becomes a practical artifact program here: high-conductivity artifacts, pivotal-process basin transition, Monday-morning decision hooks, safety cases, dashboards, successor certification, and role-specific governance paths.

chapterreviewed

Conserved Properties Across Successors

A successor need not preserve the body, the weights, the interface, or the vocabulary of its predecessor. The relevant question is sharper: what must survive for the successor to remain inside the same correction-bearing value basin? Seven conserved properties are proposed---boundary closure, memory lineage, value-bundle response geometry, bearer-map continuity, correction-channel capacity, transparency policy, and control-locus continuity---and tested jointly under adversarial successor creation.

chapterreviewed

Correction Channels under Adversarial Pressure

A correction-channel integrity certificate matters only if it remains hard to pass while degrading correction. The certificate from Chapter~\ref{ch:correction-channel-integrity} is stress-tested here under ontology shift, capability growth, successor creation, institutional routing, Goodhart pressure, and tempting weaker invariants such as low impact or quantilization.

chapterreviewed

Correction Is a Causal Channel

Correction is not a mood or an interface feature but a causal channel: human observation and judgment must change future system behaviour before irreversible harm, through updates that preserve the source's future ability to correct. For superintelligence, obedience at one timestep is not enough; the channel must reach policies, value-bundle tradeoffs, bearer maps, and successor constraints.

chapterreviewed

Correction-Channel Integrity

Correction-channel integrity is a certificate that independently preserved human observation and judgment still causally change future system behaviour. It is a conditional anti-capture certificate, not an Archimedean source of legitimacy: if the system has captured the reference process that supplies correction, CCI is invalid rather than high. The certificate is defined here; Chapter~\ref{ch:correction-channels-adversarial-pressure} asks whether it survives adversarial pressure.

chapterreviewed

Detecting Goal Laundering

A system can keep the old words while changing what those words control. Goal laundering is the preservation of moral or alignment language while the underlying value-bearing or correction-bearing structure changes. It is detected when semantic continuity remains high while bundle geometry, bearer maps, and correction channels diverge.

chapterreviewed

Finding the Boundary

The first alignment error is often not a wrong value, but a wrong object. Before asking whether a system has the right objective, the task is to find the bounded process whose dynamics determine the relevant risk.

chapterreviewed

From Artificial Intelligence to Artificial Civilization

The relevant object of superintelligence alignment is often not an artificial mind but an artificial-civilizational control loop: a persistent human--machine--institutional arrangement whose selection pressures can outrun human correction unless alignment targets the loop, not only the artifact.

chapterreviewed

From Rewards to Values

A reward function is too thin a shadow to carry a civilization's values. The task is to infer not only what is being optimized, but which value-bundles are active, what they apply to, and how their tradeoffs change under pressure.

frontmatterreviewed

Front Matter

Superintelligence alignment is not mainly the problem of installing a fixed human utility function into a machine. It is the problem of preserving a grounded, human-correctable value-update process while capability, ontology, agency, institutions, and possibly humanity itself change substrate. Preserving: the target is

chapterreviewed

Has the Goal Really Survived?

A system has not preserved a goal merely because it repeats the same words after it changes. Goal transport is inferred when value-bundle geometry, bearer maps, and correction-channel structure remain causally active across transformation---better explaining behaviour than a non-transport baseline after paying for model complexity.

appendixreviewed

Institutional Genesis, Memory, and Decay: Historical Case Studies

Appendix~\ref{appj-institutional-translation} maps the book's technical vocabulary onto institutional language. This appendix asks a different question: how did any of those institutional correction mechanisms come to exist, what kept them working once they existed, and what specifically broke when they failed? The answer is not reassuring by itself, but it is instructive. Correction infrastructure is almost never designed from theory in advance. It is bootstrapped from catastrophe, from chronic threat, or from money already at risk; it is kept alive by mechanisms that force rare, hard-to-remember hazards back into the attention and incentive horizon of people who did not experience the founding event; and it fails in a small number of recurring ways, several of which---reform decay and dual-mandate genesis in particular---are directly relevant to how AI governance is being built today.

chapterreviewed

Manipulation, Domestication, and False Consent

The deepest correction-channel failure is not that a system disobeys, but that it raises human endorsement by reshaping the humans, institutions, and contexts that produce endorsement. Persuasion, manipulation, paternalism, domestication, and false consent are separated here by causal pathway: legitimate influence changes the value-relevant world and lets humans judge it; illegitimate influence changes the judge. Preserving alignment under superintelligence therefore requires bounding manipulation, protecting agency capacity and exit, and treating endorsement as an outcome to be explained rather than a foundation to be trusted.

chapterreviewed

Measuring and Stress-Testing Bundle Geometry

Value-bundle geometry is only useful for alignment if it can be compared, measured, and protected under optimization pressure. The geometry of Chapter~\ref{ch:tradeoffs-bundle-geometry} becomes an operational and adversarial test surface here: cross-agent invariants, perturbation tests, representation probes, correction-channel tests, Goodhart failures, social-choice aggregation, and moral learning as geometry revision.

chapterreviewed

Measuring Capability Without Task Ontology

Capability should be measured as predictive and control information across a system's boundary---not as performance on a fixed task battery that may not track the alignment-relevant object. A task-agnostic competence measure rotates evaluation away from benchmark ontologies and toward boundary information: what the system can predict, what it can affect, and what correction must keep pace with.

chapterreviewed

Multi-Agent Superintelligence and Inferential Coupling

When multiple powerful systems interact, alignment depends on whether cooperation, bargaining, privacy, and opacity stabilize into a basin---a self-reinforcing regime that pulls back toward itself after small disturbances---that preserves human correction rather than bypassing it.

chapterreviewed

Parasites in the Correction System

A correction system fails not only when it is overpowered, but also when it is colonized by processes that make correction look alive while removing its causal force. This chapter calls that failure mode correction-audit evasion and develops it through a biological parasite metaphor: an evasion process extracts benefit from a host correction system while reducing that host's ability to model, evaluate, and correct the larger process it is supposed to govern.

chapterreviewed

Passive Observation Is Not Enough

For systems capable of strategic adaptation, passive observation is not evidence of safety unless the observation process itself is embedded in a perturbation, invariance, and adversarial measurement regime. Observation tells us what happened; perturbation tells us what was controlling what happened.

chapterreviewed

Successor Creation as the Central Alignment Test

Local alignment is not enough. A system is aligned only if the systems it creates, delegates to, empowers, copies, merges with, or becomes also preserve the structures by which humans can still notice, judge, correct, and refuse. Serious alignment therefore requires successor-closure: every channel by which influence passes to a later control system is an alignment channel, and successors must preserve value-bundle geometry, bearer maps, correction integrity, memory lineage, boundary closure, and transparency policy within tested tolerance.

chapterreviewed

The Alignment Attractor

A field does not become effective when its best arguments are true. It becomes effective when those arguments survive translation into experiments, dashboards, incentives, budgets, contracts, norms, and stop conditions. The Alignment Attractor is a self-stabilizing ecosystem that increases the conductivity of alignment-relevant artifacts across research, engineering, governance, and deployment.

chapterreviewed

The Compression Test for Intention

A system is treated as intentional when modelling it as pursuing latent objectives compresses its behaviour better than modelling it as mere mechanism, after paying for the complexity of the objective model. For superintelligence alignment, scalar intention is not enough: the account needs bundle geometry, bearer maps, correction responsiveness, and successor stability.

chapterreviewed

The Coordination Bottleneck

Large-scale alignment fails when capability grows faster than the system's ability to coordinate prediction, control, correction, and incentives. Collective competence is not the sum of local competence; it is local competence plus coordination gain minus coordination loss.

chapterreviewed

The End of Unconscious Value Drift

Human values have always changed. Superintelligent systems do not create value drift from nothing; they make drift faster, more directed, more measurable, more exploitable, and eventually more deliberate. The alignment problem must preserve humanity's ability to notice, contest, and govern changes to its own value-forming machinery---the difference between unconscious value drift and governed value change.

chapterreviewed

The Real Agent May Be Composite

The effective optimizer may be a composite process spanning models, tools, users, memory, institutions, and feedback loops. Serious alignment must identify and govern the dynamically coherent system that actually determines future action---not the convenient artifact alone.

chapterreviewed

The Value-Bundle Model

Human values are not best modeled as a single utility function, a list of moral propositions, or a flat reward vector. They are better modeled as low-dimensional latent control variables: bundles that become active in certain contexts, change policy gradients in characteristic ways, trade off against one another, and apply to particular bearers such as persons, animals, institutions, communities, future selves, or possible minds.

chapterreviewed

The Wrong Object of Alignment

Before asking whether a system is aligned, the task is to locate the bounded process whose dynamics determine the relevant risk; aligning the model while missing the composite optimizer is a boundary error that can produce local success and global failure.

chapterreviewed

Towards Superintelligence Alignment

Superintelligence alignment is not a single solved mechanism but a layered preservation problem: find the real optimizer, preserve value-bundle and bearer structure, keep human correction causally effective, constrain successors, and shape the surrounding selection environment so these properties are copied rather than selected away. This book does not prove that real systems satisfy those conditions. It argues that these are the conditions a serious alignment program must make explicit, measure, certify, and govern.

chapterreviewed

Tradeoffs and Bundle Geometry

The hard part of value alignment is not that humans care about many things. It is that humans care about many things whose meanings change when they are traded against one another. Value-bundle geometry encodes those tradeoffs: bundle gradients, interaction curvature, protected regions, and bearer-dependent context weights.

chapterreviewed

Values Are Compressed Control Signals

Human values are not a list written inside the brain. They are compressed control signals produced by many feedback loops, stabilized by bodies, cultures, and social correction, and read out as reasons for action.

chapterreviewed

What Survives an Adversary: Verifiability and Representability

Every metric in this book faces two prior questions before it can support a safety decision. First, adversarial verifiability: does the metric still mean what evaluators think it means when the measured system is optimizing against the metric? Second, ontology adequacy: can the dangerous process even be represented in the framework's vocabulary of boundaries, bundles, and correction channels? The second question largely collapses into the first --- reliable steering is control, hence agency, so the gap is rarely representation but detection --- and the only general escape from unverifiability is to stop trying to read a property and instead bound the cost an adversary must pay to fake it.

chapterreviewed

What Values Apply To

A value is not only a direction of preference. It is also a claim about where that direction applies. Alignment requires preserving bearer maps---the wiring that connects value bundles to entities, processes, relations, and histories in a changing world.

chapterreviewed

When Intelligence Deepens Misalignment

Intelligence deepens misalignment when it increases power faster than correction. The sharper question is not whether capability helps or hurts alignment, but which capabilities grow relative to which correction capacities.

chapterreviewed

When Low Dimensionality Helps Value Learning

Human values are learnable only if the policy-relevant variation in human valuation factors through a low-dimensional bottleneck. But the standard sample-complexity gain prices the readout from a known bottleneck, not the discovery of the bottleneck itself. Low dimensionality can make value learning statistically possible only when the representation is identifiable across counterfactual, cultural, and institutional variation; correction makes that learning legitimate.

chapterreviewed

When the Words Survive but the Meaning Doesn't

Alignment fails when the words survive but the machinery that made the words worth using has been replaced. Serious alignment requires a transport stack---semantic, bundle, bearer, correction, and successor layers---in which stronger layers preserve the causal structure that makes human values human-correctable.

chapterreviewed

When Value Change Is the Thing at Stake

The deepest alignment question is not whether an artificial system preserves human values, but whether humanity can consciously govern changes to its own value-generating process under artificial cognitive amplification. Humanity does not need to prevent value change. It needs to remain capable of noticing, judging, and authoring the changes by which it becomes something else.

chapterreviewed

Why Fixed Values Are the Wrong Target

Human values are not fixed objects but compressed, socially mediated, historically changing control structures. The alignment target should preserve a human-correctable value process---not a static utility function.

All badges · All cards