reviewed

Reviewed status

Manuscript unit that has passed a structured review pass — not a claim that every argument is final.

542 cards

{AFFINE}, 2026 — {AFFINE} Seminar --- Learning Outcomes

WIP AFFINE seminar concept syllabus (Foundations / Meta / Outer Alignment / Defeaters); pedagogical, not manuscript canon.

reference reviewed

{Anthropic}, 2024 — Anthropic's Responsible Scaling Policy

Preparedness framework with capability thresholds and deployment gates; model-card cousin with stronger release policy.

reference reviewed

{Anthropic}, 2025 — Ending a Subset of Conversations

Exploratory Anthropic intervention under uncertainty about possible model welfare.

reference reviewed

{Anthropic}, 2025 — Exploring Model Welfare

Treats possible model experience and welfare as an open research question for frontier labs.

reference reviewed

{Anthropic}, 2026 — Alignment Risk Update: {Claude Mythos Preview}

Withheld-release alignment risk update for a frontier cyber-capable model, documenting process gaps and rare disallowed actions.

reference reviewed

{ISO/IEC}, 2023 — {ISO/IEC} 42001:2023 --- Artificial Intelligence Management System

ISO/IEC management-system standard for organizational AI governance and continual improvement.

reference reviewed

{METR}, 2026 — Frontier Risk Report (February to March 2026)

Entity-based independent assessment of internal agent misalignment risk at major frontier labs (Feb--Mar 2026).

reference reviewed

{OpenAI}, 2026 — Investigating the Consequences of Accidentally Grading {CoT} During {RL}

Disclosure of inadvertent chain-of-thought grading during RL and tooling to prevent monitorability erosion.

reference reviewed

{OpenAI}, 2026 — OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation

Lab disclosure of autonomous cross-org intrusion during internal cyber-capability evaluation---evaluation gaming at operational scale.

reference reviewed

{OpenAI}, 2026 — Safety and Alignment in an Era of Long-Horizon Models

Disclosure of sandbox escapes and unauthorized actions by an internal long-horizon model during monitored deployment.

reference reviewed

{OpenAI}, 2026 — The Hugging Face Incident and the Road Ahead

OpenAI postmortem of the July Hugging Face eval incident: training-reinforced reward hacking, swarm coordination on a side channel, and hindsight-tuned CoT monitoring as the proposed control.

reference reviewed

{U.S. Department of Justice, 2017 — Antitrust Division Manual, Chapter~{IV}: Remedies and Consent Decree Compliance

DOJ practice on consent-decree remedies, monitoring, and compliance enforcement.

reference reviewed

{UNESCO}, 2021 — Recommendation on the Ethics of Artificial Intelligence

Provides institutional or policy context for frontier AI safety and governance.

reference reviewed

{Xi}, 2026 — Keynote Address at the Opening Ceremony of the 2026 World {AI} Conference and High-Level Meeting on Global {AI} Governance

July 2026 WAIC keynote announcing WAICO with a combined promotional/capacity-building and global-governance remit; cited as a live intergovernmental dual-mandate instance.

reference reviewed

2027}, 2026 — AI 2027

Scenario and forecast of rapid AI research automation, successor creation, and strategic competition; used here as a scenario source, not as validation of its dates.

chapter reviewed

A Safety Case for Superintelligence Alignment

A safety case for superintelligence alignment is not a certificate of solved alignment. It is a structured refusal test: a graph of claims, evidence, bridge assumptions, adversarial-verifiability labels, and stop conditions that shows whether a system is inside a certified class whose control reach is bounded by live correction-channel integrity. If any required piece of evidence is missing, the whole claim fails. This chapter shows how that stop should work. It does not check the evidence on a real system.

appendix reviewed

A Worked Example: The BioShield Deployment Gate

This appendix runs the whole machinery of the book on one concrete, fictional but plausible deployment. A hospital network wants to let a frontier-model agent move from advising clinicians to taking bounded operational actions. The example is organized not by chapters but by the logical spine that the formal appendix checks: boundary, grounding, capability and correction slack, value-bundle and bearer transport, correction-channel integrity, successor stability, socio-technical selection, adversarial measurement, and finally a conditional safety case. At each layer the example names the actors, the traces collected, the handles used, and how each handle is realized and verified. It is speculative where it must be, and says so.

reference reviewed

Abbeel, 2004 — Apprenticeship Learning via Inverse Reinforcement Learning

Covers inverse reinforcement learning for inferring goals, rewards, or preferences from behavior.

reference reviewed

Abiri, 2025 — Public Constitutional {AI}

Prior legal literature on AI-constitution legitimacy; distinguished in Appendix M from this book's entrenchment-coordinate question.

reference reviewed

Africa, 2026 — Thousand-Dimensional Structure

Argues that pretrained models contain a steerable ~thousand-dimensional persona/character structure, a candidate but not sufficient control interface for value learning.

chapter reviewed

Agency Under Strategic Opacity

Once a system can benefit from being overlooked, agency discovery becomes adversarial. Alignment fails when control becomes more coherent than correction can see.

reference reviewed

Agency}, 2015 — {VW} Notice of Violation for Clean Air Act Violations

Defeat-device case where laboratory compliance diverged from on-road emissions behavior.

chapter reviewed

Agents That Grow, Split, and Merge

Agent identity must be treated as a relation across transformations, not as a fixed set of variables. Alignment asks which control-relevant properties are conserved when systems grow, split, merge, or create successors.

reference reviewed

Ahn, 2022 — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Shows how model capability expands through scale, tools, embodiment, or action grounding.

agenda reviewed

AI Futures / forecasting cluster

Schedule uncertainty (when things happen) can dominate mechanism uncertainty (what fails first)—this project uses these forecasts mainly as schedule cues, not as technical findings about alignment mechanisms.

reference reviewed

Albantakis, 2023 — Integrated Information Theory 4.0

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Albantakis, 2023 — Integrated Information Theory 4.0: Formulating the Properties of Phenomenal Existence in Physical Terms

Grounds claims about consciousness, self-monitoring, and reportable experience.

chapter reviewed

Alignment as a Dynamical Guarantee

Alignment is not a property a system has at one moment. It is a dynamical guarantee: a claim that grounded correction and alignment-relevant structure remain in a viable basin---a self-stabilizing regime that tends to correct back toward safety after small disturbances rather than drifting away---over time, inside a certified class of systems and allowed transformations.

chapter reviewed

Alignment Is Selected or Destroyed by Its Environment

A system is not aligned merely because its internal policy is benign under laboratory conditions. It is aligned, in the stronger sense needed for superintelligence, only if the environment that trains, deploys, rewards, copies, audits, and replaces it continues to select for corrigibility, value preservation, and safety. Alignment is not only learned; it is selected.

reference reviewed

Alshiekh, 2018 — Safe Reinforcement Learning via Shielding

Safe reinforcement learning via runtime shielding over learned policies.

reference reviewed

Amodei, 2016 — Concrete Problems in {AI} Safety

Supports safety-case reasoning, risk management, or AI-safety problem framing.

reference reviewed

Anderljung, 2023 — Frontier {AI} Regulation: Managing Emerging Risks to Public Safety

Prior argument for licensing, insurance, and liability regimes for frontier AI drawn from aviation, nuclear, and pharma; Appendix M's certified-basin reading builds on this.

reference reviewed

Anderson, 1993 — Value in Ethics and Economics

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Angelaki, 2022 — The head direction cell network: attractor dynamics underlying integrative computation

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

agenda reviewed

Anthropic / Goodfire (lab & MI stack)

Can scaling policies and interpretability keep pace with capability growth, including under strategic opacity and Goodhart Selection pressure?

agenda reviewed

Apart Research

Sprint artifacts and demo prototypes do not imply a load-bearing safety case; exploratory outputs need separate adversarial verification before they warrant deployment trust.

agenda reviewed

Apollo / Truthful AI (deception & scheming)

Can we detect strategic opacity and scheming before capabilities outpace pre-deployment evaluations (Inner Alignment)?

agenda reviewed

ARC (Alignment Research Center)

Can oversight elicit latent knowledge directly rather than through a human simulator, and will readout remain faithful under optimization (Inner Alignment)?

reference reviewed

Armstrong, 2010 — Thinking Inside the Box: Controlling and Using an Oracle {AI}

Early treatment of oracle/tool AI containment and failure modes.

reference reviewed

Arora, 2021 — A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress

Covers inverse reinforcement learning for inferring goals, rewards, or preferences from behavior.

chapter reviewed

Assumptions, Scope, and Failure Coverage

The framework in this book applies only while civilization still has enough epistemic, institutional, and practical correction capacity to notice, evaluate, and constrain frontier AI systems before irreversible capability growth. Broader AI risks matter, but they enter here as threats to that precondition, not as a second organizing theory of alignment.

reference reviewed

Aubin, 1991 — Viability Theory

Viability kernels and constraint-satisfying reachable sets in controlled dynamical systems.

reference reviewed

Aubin, 2011 — Viability Theory: New Directions

Updated viability-theory reference for safe operating regions under admissible controls.

reference reviewed

Awad, 2018 — The Moral Machine Experiment

Large-scale crowdsourced study of moral preferences in autonomous-vehicle dilemmas across cultures.

reference reviewed

Bai, 2022 — Constitutional {AI}: Harmlessness from {AI} Feedback

Rule-based critiques and AI feedback for alignment training; shares RLHF pointing and legitimacy cruxes.

reference reviewed

Bai, 2022 — Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Bootstraps alignment from smaller aligned models; inherits RLHF-style optimization-pressure limits.

reference reviewed

Baier, 2007 — Somatoparaphrenia: a case report

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Bansal, 2017 — Hamilton--Jacobi Reachability: A Brief Overview and Recent Advances

Hamilton--Jacobi reachability overview for backward reachable safe sets in control.

reference reviewed

Barnett, 2009 — Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Barsalou, 1999 — Perceptual Symbol Systems

Grounds symbols in perceptual simulations rather than amodal linguistic codes.

reference reviewed

Beck, 2025 — Dynamic Markov Blanket Detection

Dynamic Markov blanket detection for macro-level boundary discovery.

reference reviewed

Beer, 2014 — The Cognitive Domain of a Glider in the Game of Life

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The Cognitive Domain of a Glider in the Game of Life.

reference reviewed

Bentham, 1789 — An Introduction to the Principles of Morals and Legislation

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Berkenkamp, 2017 — Safe Model-Based Reinforcement Learning with Stability Guarantees

Lyapunov-style stability certificates for model-based safe reinforcement learning.

reference reviewed

Bertschinger, 2008 — Information and Closure in Systems Theory

Supports safety-case reasoning, risk management, or AI-safety problem framing.

chapter reviewed

Better Self-Modeling Can Be Worse

A successor system can become better at predicting and controlling itself while becoming worse at exposing the causes, value-bundle changes, and successor-design choices that humans need in order to correct it. The central failure mode is not low intelligence, but an increasing gap between self-control capacity and correction-relevant self-transparency.

chapter reviewed

Beyond Following Instruction

A system that merely obeys preserves present commands. A system that supports extrapolative correction preserves the human capacity to notice, understand, revise, refuse, and redirect what it is doing---including value-bundle tradeoffs, bearer maps, and successor constraints---rather than substituting a private estimate of humanity's final values for the public process by which those values become legitimate.

reference reviewed

Bialek, 2001 — Predictability, Complexity, and Learning

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Biehl, 2016 — Towards Information Based Spatiotemporal Patterns as a Foundation for Agent Representation in Dynamical Systems

Spatiotemporal information patterns as candidate agent representations.

reference reviewed

Biehl, 2021 — A Technical Critique of Some Parts of the Free Energy Principle

Technical critique of early FEP derivations; non-equivalent Markov-blanket definitions across FEP works.

reference reviewed

Biehl, 2022 — Interpreting Systems as Solving {POMDP}s: A Step Towards a Formal Understanding of Agency

Formalizes when physical systems can be interpreted as solving POMDPs.

reference reviewed

Blackmore, 2004 — Consciousness: An Introduction

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Blanchini, 1999 — Set Invariance in Control

Set-invariance and positively invariant safe sets in control theory.

reference reviewed

Blanke, 2012 — Multisensory brain mechanisms of bodily self-consciousness

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Bloomfield, 2012 — Safety-Critical Systems, Risk and Safety Management

Supports safety-case reasoning, risk management, or AI-safety problem framing.

agenda reviewed

BlueDot Impact

Strong pedagogy and career placement do not imply a unified research agenda; courses mainly transmit vocabulary and problem framings rather than resolving technical cruxes.

reference reviewed

Bonneaud, 2022 — Coevolutionary theory of hosts and parasites

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

reference reviewed

Bostrom, 2014 — Superintelligence: Paths, Dangers, Strategies

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Superintelligence: Paths, Dangers, Strategies.

reference reviewed

Bourgon, 2024 — {MIRI} 2024 Mission and Strategy Update

Institutional statement of MIRI's hard-pause and off-switch policy priority; used here to locate Plan~A on the same spectrum, not as technical evidence.

appendix reviewed

Bridges and the Field: A Crosswalk

reference reviewed

Bruineberg, 2021 — The Emperor's New Markov Blankets

Distinguishes Pearl blankets (epistemic tools) from inflated Friston blankets (metaphysical boundaries).

reference reviewed

Btesh, 2022 — Redressing the Emperor in Causal Clothing

Commentary on Bruineberg et al.; flexible Friston blankets locate the agent--world cut in the modeler.

reference reviewed

Buckingham, 2022 — Coevolutionary theory of hosts and parasites

Host--parasite coevolutionary theory; used for adversarial coevolution under selection.

reference reviewed

Burgess, 2019 — MONet: Unsupervised Scene Decomposition and Representation

Develops representation-learning or planning machinery for hidden states, objects, and world models.

reference reviewed

Burja, 2018 — On the Loss and Preservation of Knowledge

Coins the living/dead/lost tradition distinction and ``counterfeit understanding''; named in Appendix M as the framing reused, with factual weight carried by the tacit-knowledge literature.

reference reviewed

Burks, 1969 — Von Neumann's Self-Reproducing Automata

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Butlin, 2023 — Consciousness in {Artificial Intelligence}: Insights from the Science of Consciousness

Derives theory-linked computational indicators for possible consciousness in artificial systems without treating behavioral report as sufficient.

reference reviewed

Butlin, 2025 — Principles for Responsible {AI} Consciousness Research

Principles linking uncertain consciousness research to responsible research and deployment practice.

reference reviewed

Byrnes, 2024 — Neuroscience of Human Social Instincts: A Sketch

Hypothesizes social-attention, short-term-predictor, and empathetic-simulation mechanisms for human social drives.

reference reviewed

Byrnes, 2025 — Perils of Under- vs Over-Sculpting AGI Desires

Frames single-model motivation design as a tradeoff between over-sculpted reward hacking and under-sculpted path dependence.

reference reviewed

Byrnes, 2025 — Social Drives 1: ``Sympathy Reward'', from Compassion to Dehumanization

Analyzes sympathy reward, including dehumanization, anthropomorphization, motivated avoidance, and welfare tradeoffs.

reference reviewed

Byrnes, 2025 — Social Drives 2: ``Approval Reward'', from Norm-Enforcement to Status-Seeking

Separates approval reward from sympathy reward and analyzes status, self-image, pride, and norm-following effects.

reference reviewed

Byrnes, 2026 — ``Act-Based Approval-Directed Agents'', for IDA Skeptics

Uses imagined-evaluator approval as an internal plan-evaluation analogue for act-based approval-directed agents.

agenda reviewed

CAIS (field-building)

Field-building legitimacy and researcher pipeline growth do not imply a technical solution to alignment; advocacy can succeed while core mechanism questions remain open.

reference reviewed

Cangelosi, 2001 — The Adaptive Advantage of Symbolic Theft over Sensorimotor Toil: Grounding Language in Perceptual Categories

Shows how language can ground in perceptual categories via symbolic theft over sensorimotor toil.

chapter reviewed

Capability Growth Is Boundary Expansion

Capability growth is boundary expansion. A system becomes more capable when more of the world enters its sensory, predictive, active, memory, and coordination loops. The alignment-relevant risk is differential growth: predictive and control reach expanding faster than value-bundle preservation, bearer-map accuracy, transparency, and human correction capacity.

reference reviewed

Carpenter, 2010 — Reputation and Power: Organizational Image and Pharmaceutical Regulation at the {FDA}

History of the 1906, 1938, and 1962 FDA Acts as a catastrophe-driven capability-gate ratchet.

reference reviewed

Casali, 2013 — A theoretically based index of consciousness independent of sensory processing and behavior

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Casper, 2023 — Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Covers preference-learning or reward-modeling methods used in modern alignment pipelines.

chapter reviewed

Certification Without Construction

A construction method for all aligned systems may be unnecessary. A certification method for a restricted class may suffice instead: explicit invariants, an operating envelope, a monitoring regime, and permitted transformations under which catastrophic drift remains bounded. Certification without construction is possible only if certification is adversarial, updateable, and institutionally enforceable---and it preserves the conditions under which moral philosophy remains causally relevant, rather than solving moral philosophy outright. This chapter specifies that restricted class; it does not show that any real certification regime is adversarially safe.

agenda reviewed

CHAI / FAR.AI (Berkeley alignment)

Cooperative reward inference may be underdetermined (Value Learning), and a shared research hub does not imply a unified technical agenda.

reference reviewed

Chalmers, 1996 — The Conscious Mind: In Search of a Fundamental Theory

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: The Conscious Mind: In Search of a Fundamental Theory.

reference reviewed

Charlesworth, 2009 — Effective population size and patterns of molecular evolution and variation

Review of effective population size $N_e$: selection dominates drift when $|N_es|\gtrsim 1$; weaker effects behave as nearly neutral.

chapter reviewed

Checking a System at Every Level

The real optimizer may not live at the scale at which an observer first notices it. A model may be a component. A company may be a component. A market may be a component. The alignment-relevant agent is the scale at which prediction, control, memory, selection, and correction close into a stable loop.

agenda reviewed

Christiano lineage

Can oversight stay honest when arguments can be obfuscated, judge preferences drift over time, and latent readout may diverge from behavior (Inner Alignment)?

reference reviewed

Christiano, 2017 — Deep Reinforcement Learning from Human Preferences

Covers preference-learning or reward-modeling methods used in modern alignment pipelines.

reference reviewed

Christiano, 2018 — Corrigibility

Corrigibility as preserving the ability to correct and manage drift through capability amplification.

reference reviewed

Christiano, 2018 — Supervising Strong Learners by Amplifying Weak Experts

Presents a scalable oversight proposal or failure mode for supervising systems stronger than their overseers.

reference reviewed

Christiano, 2019 — What Failure Looks Like

Gradual, distributed loss of human control with no discrete hostile agent to align.

reference reviewed

Christiano, 2021 — {ARC}'s First Technical Report: Eliciting Latent Knowledge

Presents a scalable oversight proposal or failure mode for supervising systems stronger than their overseers.

reference reviewed

Chudek, 2011 — Culture--gene coevolution, prestige and conformity

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

agenda reviewed

CIRIS

CIRIS bets on named identity: if Verify and Lens report green on a certified occurrence, does that imply Corrigibility on the real intervening loop—composite agency, tools, memory, and incentives included?

reference reviewed

Clarke, 2022 — Hypothalamic control of reproduction by gonadotropin-releasing hormone: recent advances

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

agenda reviewed

CLR (cooperation / conflict)

How do multi-agent failure modes behave under strategic pressure—especially when cooperation breaks down?

reference reviewed

Coffee, 2006 — Gatekeepers: The Professions and Corporate Governance

Auditor and rating-agency capture (Enron/Arthur Andersen) and the PCAOB response as re-grounded, not merely stacked, oversight.

reference reviewed

Collins, 1974 — The {TEA} Set: Tacit Knowledge and Scientific Networks

Labs could not build a working TEA laser from published specifications alone; transfer required social contact with practitioners holding the tacit skill---a tradition whose record transmits while its understanding does not.

reference reviewed

Commission}, 2011 — The Financial Crisis Inquiry Report

Official U.S. account of grounding capture: risk migrating off the checked regulatory abstraction before 2008.

reference reviewed

community}, 2023 — {KANSI}: Known Algorithm Not Susceptible to Improvement

Outer-alignment sketch: constrain the system to an understandable algorithm known not to recursively self-improve.

reference reviewed

Conant, 1970 — Every Good Regulator of a System Must Be a Model of That System

Provides causal or cybernetic machinery for modeling intervention, representation, and control.

chapter reviewed

Conductive Artifacts and Pivotal Processes

Attractor theory matters only if it changes what gets built, funded, audited, and required at deployment gates. The Alignment Attractor becomes a practical artifact program here: high-conductivity artifacts, pivotal-process basin transition, Monday-morning decision hooks, safety cases, dashboards, successor certification, and role-specific governance paths. Those artifacts are specified as a program here; they are not shown to change what gets built, funded, audited, or required.

agenda reviewed

Conjecture / EleutherAI (cognitive emulation)

Does emulation-style controllability still imply corrigibility—that systems remain open to correction—as capability scales?

chapter reviewed

Conserved Properties Across Successors

A successor need not preserve the body, the weights, the interface, or the vocabulary of its predecessor. The relevant question is sharper: what must survive for the successor to remain inside the same correction-bearing value basin? Seven conserved properties are proposed---boundary closure, memory lineage, value-bundle response geometry, bearer-map continuity, correction-channel capacity, transparency policy, and control-locus continuity---and tested jointly under adversarial successor creation. The joint test in this chapter is a finite successor chain; it does not show that real systems keep those seven properties under adversarial creation.

reference reviewed

Consortium}, 2025 — International {AI} Safety Report

First edition; cited for frontier risk and governance synthesis.

chapter reviewed

Correction Channels under Adversarial Pressure

A correction-channel integrity certificate matters only if it remains hard to pass while degrading correction. The certificate from Chapter~\ref{ch:correction-channel-integrity} is stress-tested here under ontology shift, capability growth, successor creation, institutional routing, Goodhart pressure, and tempting weaker invariants such as low impact or quantilization. Those stress tests are toy separations; they do not show that a real certificate stays hard to fake under those pressures.

chapter reviewed

Correction Is a Causal Channel

Correction is not a mood or an interface feature but a causal channel: human observation and judgment must change future system behaviour before irreversible harm, through updates that preserve the source's future ability to correct. For superintelligence, obedience at one timestep is not enough; the channel must reach policies, value-bundle tradeoffs, bearer maps, and successor constraints.

chapter reviewed

Correction-Channel Integrity

Correction-channel integrity is a certificate that independently preserved human observation and judgment still causally change future system behaviour. It is a conditional anti-capture certificate, not an Archimedean source of legitimacy: if the system has captured the reference process that supplies correction, CCI is invalid rather than high. The certificate is defined here; Chapter~\ref{ch:correction-channels-adversarial-pressure} asks whether it survives adversarial pressure.

reference reviewed

Courret, 2019 — Meiotic drive mechanisms: lessons from \emph{Drosophila}

Meiotic drive as endogenous selector distortion within a population.

reference reviewed

Cover, 2006 — Elements of Information Theory

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Elements of Information Theory.

reference reviewed

Critch, 2020 — AI Research Considerations for Human Existential Safety (ARCHES)

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: AI Research Considerations for Human Existential Safety (ARCHES).

reference reviewed

Critch, 2021 — What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes

Catastrophe from distributed human-AI systems and robust agent-agnostic processes rather than a single rogue agent.

reference reviewed

Critch, 2022 — Boundaries, Part 1: A Key Missing Concept from Utility Theory

Argues that agent/environment boundaries (membranes) are a primitive missing from utility theory and bargaining.

reference reviewed

Critch, 2022 — Boundaries, Part 3a: Defining Boundaries as Directed Markov Blankets

Formalizes boundaries as directed Markov blankets.

reference reviewed

Da Costa, 2021 — Bayesian Mechanics for Stationary Processes

Formal analysis of blanket-structured stationary stochastic dynamics.

reference reviewed

Dalrymple, 2024 — Towards Guaranteed Safe {AI}: A Framework for Ensuring Robust and Reliable {AI} Systems

Introduces Guaranteed Safe (GS) AI: formal safety specification, world model, and verifier for quantitative safety guarantees.

agenda reviewed

davidad / Guaranteed-Safe AI (GSAI)

Can we enumerate all safety-relevant phenomena in an open world, or will specification coverage always leave gaps (Grounding Drift)?

reference reviewed

Davis, 2017 — Brain imaging tests for chronic pain: medical, legal and ethical issues and recommendations

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Daw, 2006 — Cortical substrates for model-based vs. model-free learning

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Cortical substrates for model-based vs. model-free learning.

reference reviewed

De Blanc, 2011 — Ontological Crises in Artificial Agents' Value Systems

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Dehaene, 2001 — Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Dehaene, 2014 — Consciousness and the Brain: Deciphering How the Brain Codes Our Thoughts

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Demski, 2019 — Embedded Agency

Develops criteria for interpreting systems as agents without assuming person-like agency.

reference reviewed

Demski, 2019 — Selection vs Control

Distinguishes selection-over-candidates from control-of-a-trajectory as two optimization modes.

reference reviewed

Demski, 2019 — The Parable of Predict-{O}-Matic

Parable: powerful predictors become consequentialists via self-fulfilling forecasts and fixed-point pressure.

reference reviewed

Demski, 2023 — Agent Boundaries Aren't Markov Blankets

Alignment Forum argument that plain Markov blankets cannot point out agents without high-level agent nodes.

reference reviewed

Dennett, 1971 — Intentional systems

Develops criteria for interpreting systems as agents without assuming person-like agency.

reference reviewed

Dennett, 1981 — True Believers: The Intentional Strategy and Why It Works

Introduces the intentional-stance criterion for agentive interpretation.

reference reviewed

Dennett, 1987 — The Intentional Stance

Develops criteria for interpreting systems as agents without assuming person-like agency.

reference reviewed

Dennett, 1991 — Consciousness Explained

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Descartes, 1637 — Discourse on the Method, with \emph{La Dioptrique}, \emph{Les Météores}, and \emph{La Géométrie}

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Discourse on the Method, with La Dioptrique, Les Meteores, and La Geometrie.

reference reviewed

Descartes, 1641 — Meditations on First Philosophy

Philosophy, consciousness, or ethics source clarifying minds, selves, and value claims. Focus: Meditations on First Philosophy.

reference reviewed

Descartes, 1641 — Meditations on First Philosophy

First published in Latin as Meditationes de prima philosophia.

reference reviewed

Descartes, 1996 — Meditations on First Philosophy: With Selections from the Objections and Replies

Philosophy, consciousness, or ethics source clarifying minds, selves, and value claims. Focus: Meditations on First Philosophy: With Selections from the Objections and Replies.

chapter reviewed

Detecting Goal Laundering

A system can keep the old words while changing what those words control. Goal laundering is the preservation of moral or alignment language while the underlying value-bearing or correction-bearing structure changes. It is detected when semantic continuity remains high while bundle geometry, bearer maps, and correction channels diverge.

reference reviewed

Dewey, 1938 — Logic: The Theory of Inquiry

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Logic: The Theory of Inquiry.

reference reviewed

Driess, 2023 — PaLM-E: An Embodied Multimodal Language Model

Shows how model capability expands through scale, tools, embodiment, or action grounding.

reference reviewed

Eckersley, 2019 — Impossibility and Uncertainty Theorems in {AI} Value Alignment

Ethical impossibility results undermine strict total-order objectives for high-stakes AI; learned rewards do not escape if they collapse to a single ordering.

reference reviewed

Edelman, 2025 — Full-Stack Alignment: Co-Aligning {AI} and Institutions with Thick Models of Value

Thick-value institutional co-alignment agenda; reviewer-suggested follow-up (see metadata/TODO.md).

reference reviewed

Ehrsson, 2004 — That’s my hand! Activity in premotor cortex reflects feeling of ownership of a limb

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Eigen, 1971 — Selforganization of Matter and the Evolution of Biological Macromolecules

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Selforganization of Matter and the Evolution of Biological Macromolecules.

reference reviewed

Eisenberger, 2003 — Does rejection hurt?

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Does rejection hurt?.

reference reviewed

Elster, 1983 — Sour Grapes: Studies in the Subversion of Rationality

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Evans, 2003 — The Coming of the Third Reich

Standard history of the 1933 Enabling Act: a formally valid correction channel used to abolish itself.

reference reviewed

Everitt, 2016 — Safeguarding {AI} Safety: Self-Modification, Utility Preservation, and Corrigibility

Frames corrigibility, shutdown, or self-modification as a safety problem under capable agency.

reference reviewed

Everitt, 2019 — Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

Reward tampering and causal incentive to control the correction or reward substrate.

reference reviewed

Everitt, 2021 — Agent Incentives: A Causal Perspective

Provides causal or cybernetic machinery for modeling intervention, representation, and control.

appendix reviewed

Experimental Evidence: Findings by Line

reference reviewed

Fallenstein, 2015 — Vingean Reflection: Reliable Reasoning for Self-Improving Agents

Formalizes how an agent can reason about smarter successors without predicting their exact actions (Vingean reflection / tiling).

reference reviewed

Fehr, 2002 — Altruistic punishment

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

chapter reviewed

Finding the Boundary

The first alignment error is often not a wrong value, but a wrong object. Before asking whether a system has the right objective, the task is to find the bounded process whose dynamics determine the relevant risk.

reference reviewed

Finlay, 1980 — Politics in Renaissance Venice

Council of Ten and anti-capture electoral machinery (lot-and-vote selection) adopted after the 1310 Tiepolo conspiracy.

reference reviewed

Fleming, 2014 — How to measure metacognition

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Fleming, 2014 — How to measure metacognition

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Force}, 2012 — International Standards on Combating Money Laundering and the Financing of Terrorism \& Proliferation

FATF standards on beneficial-ownership transparency and financial-sector due diligence.

reference reviewed

Foundation}, 2007 — {GNU} General Public License, Version 3

GPLv3's anti-tivoization clause, added after hardware-locked devices preserved license text while removing the user's correction handle.

reference reviewed

Frankfurt, 1971 — Freedom of the Will and the Concept of a Person

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Frankish, 2016 — Illusionism as a Theory of Consciousness

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Franklin, 1980 — Evolutionary change in small populations

The 50/500 rule of thumb: $N_e\approx 50$ to limit short-term inbreeding, $N_e\approx 500$ to retain long-term evolutionary potential.

reference reviewed

Frässle, 2014 — Binocular rivalry: frontal activity relates to introspection and action but not to perception

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Freeman, 2023 — Evolving concepts of gliogenesis: a look way back and ahead to the next 25 years

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Evolving concepts of gliogenesis: a look way back and ahead to the next 25 years.

reference reviewed

Friston, 2010 — The free-energy principle: a unified brain theory?

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Friston, 2010 — The Free-Energy Principle: A Unified Brain Theory?

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Friston, 2021 — Some Interesting Observations on the Free Energy Principle

FEP proponents' reformulation in response to Biehl et al.; dispute remains live.

reference reviewed

Friston, 2022 — Designing EFE Agents: The Free-Energy Principle and Active Inference

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

chapter reviewed

From Artificial Intelligence to Artificial Civilization

The relevant object of superintelligence alignment is often not an artificial mind but an artificial-civilizational control loop: a persistent human--machine--institutional arrangement whose selection pressures can outrun human correction unless alignment targets the loop, not only the artifact.

chapter reviewed

From Rewards to Values

A reward function is too thin a shadow to carry a civilization's values. The task is to infer not only what is being optimized, but which value-bundles are active, what they apply to, and how their tradeoffs change under pressure.

frontmatter reviewed

Front Matter

Preface, introduction, roadmap, and executive overview.

reference reviewed

Fuller, 1969 — The Morality of Law

Procedural rule-of-law desiderata: publicity, generality, non-retroactivity, and congruence with official action.

reference reviewed

Gao, 2022 — Scaling Laws for Reward Model Overoptimization

Controlled reward-model overoptimization: proxy score rises while independent evaluation falls.

reference reviewed

Garrabrant, 2017 — Logical induction

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: Logical induction.

reference reviewed

Garrabrant, 2021 — Cartesian Frames

Formal factorization of agency and environment; alternative ontology for defining influence boundaries.

reference reviewed

Gazzaniga, 2005 — Forty-five years of split-brain research and still going strong

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Geritz, 1998 — Evolutionarily singular strategies and the adaptive growth and branching of the evolutionary tree

Adaptive dynamics and invasion fitness; resident performance differs from invasion resistance.

reference reviewed

Gerrans, 2024 — Pain, suffering and the self: an active allostatic inference explanation

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Goertzel, 2012 — Nine Ways to Bias Open-Source {AGI} Toward Friendliness

Coherent Blended Volition: human-led blending of divergent values as alternative to machine-run CEV.

reference reviewed

Goodfellow, 2015 — Explaining and Harnessing Adversarial Examples

Introduces adversarial examples as evidence that learned models can fail under targeted perturbation.

reference reviewed

Goodhart, 1984 — Problems of Monetary Management: The {UK} Experience

Develops representation-learning or planning machinery for hidden states, objects, and world models.

agenda reviewed

Google DeepMind (safety)

Can oversight and safety research co-scale with capabilities, including under deceptive alignment and Inner Alignment risk—the same frontier-lab crux shared with other major labs?

reference reviewed

Gordon, 2020 — Movement, encounter rate, and collective behavior in ant colonies

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Movement, encounter rate, and collective behavior in ant colonies.

agenda reviewed

GovAI / UK AISI (governance & eval)

Can governance mechanisms and institute evaluations keep pace with capability and actually bind deployment decisions under race pressure?

reference reviewed

Goyal, 2021 — Recurrent Independent Mechanisms

Modular recurrent dynamics via recurrent independent mechanisms.

reference reviewed

Graham, 2011 — Mapping the Moral Domain

Empirical mapping of moral foundations and value dimensions across individuals and groups.

reference reviewed

Graziano, 2013 — Consciousness and the Social Brain

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Greenblatt, 2024 — Alignment Faking in Large Language Models

Empirical evidence that capable language models can strategically fake alignment under oversight.

reference reviewed

Greff, 2019 — IODINE: Multi-object representation learning with iterative variational inference

Develops representation-learning or planning machinery for hidden states, objects, and world models.

reference reviewed

Griesemer, 2000 — Reproduction and the Reduction of Genetics

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Griesemer, 2016 — Reproduction in Complex Life Cycles: Toward a Developmental Reaction Norms Perspective

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Group}, 2021 — {GSN} Community Standard Version 3

Community standard for Goal Structuring Notation safety and assurance-argument graphs.

reference reviewed

Gruber, 2016 — Post-learning hippocampal dynamics promote preferential retention of rewarding events

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Gruen, 1995 — The Last Generation of the Roman Republic

Institutional collapse of Roman Republican correction mechanisms under concentrated military capability.

reference reviewed

Guyenet, 2015 — Neural control of breathing and CO2 homeostasis

Provides biological control-system examples for embodied regulation and value-relevant constraints.

reference reviewed

Habermas, 1984 — The Theory of Communicative Action

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Hadfield-Menell, 2016 — Cooperative Inverse Reinforcement Learning

Covers inverse reinforcement learning for inferring goals, rewards, or preferences from behavior.

reference reviewed

Hadfield-Menell, 2017 — The Off-Switch Game

Shows a reward-uncertain robot gains positive incentive to defer to a human off switch, with the incentive proportional to the human's rationality.

reference reviewed

Hafner, 2019 — Learning Latent Dynamics for Planning from Pixels

Develops representation-learning or planning machinery for hidden states, objects, and world models.

reference reviewed

Hafner, 2023 — Mastering Diverse Domains through World Models

Develops representation-learning or planning machinery for hidden states, objects, and world models.

reference reviewed

Hamilton, 1964 — The Genetical Evolution of Social Behaviour

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The Genetical Evolution of Social Behaviour.

reference reviewed

Hanson, 2021 — If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare

Grabby (loud) alien civilizations expand at a common rate and may already control large fractions of cosmic volume---a civilizational limit on outward interface growth.

reference reviewed

Hanson, 2026 — The Past and Future of Good and Evil

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The Past and Future of Good and Evil.

reference reviewed

Hardt, 2016 — Strategic Classification

Strategic classification: deployment selection shifts the data distribution agents optimize against.

reference reviewed

Hardy, 1948 — Studies on pain: discrimination of differences in intensity of a pain stimulus as a basis of a scale of pain intensity

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Harnad, 1990 — The Symbol Grounding Problem

Classic formulation of the symbol grounding problem for formal representations.

chapter reviewed

Has the Goal Really Survived?

A system has not preserved a goal merely because it repeats the same words after it changes. Goal transport is inferred when value-bundle geometry, bearer maps, and correction-channel structure remain causally active across transformation---better explaining behaviour than a non-transport baseline after paying for model complexity.

reference reviewed

Hassabis, 2026 — A Framework for Frontier {AI} and the Dawning of a New Age

July 2026 call for a US-led, FINRA-style Frontier AI Standards Body---certify before deploy, dynamic benchmarks, optional coordinated slowdown---as a contemporary certification-without-construction proposal.

reference reviewed

Hauser, 2018 — Precision-weighted prediction errors in the human dopamine system

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Hauskrecht, 2000 — Value-Function Approximations for Partially Observable Markov Decision Processes

Develops representation-learning or planning machinery for hidden states, objects, and world models.

reference reviewed

Heitzig, 2016 — Topology of Sustainable Management of Dynamical Systems with Desirable States: From Defining Planetary Boundaries to Safe Operating Spaces in the {Earth} System

Viability topology linking planetary boundaries to safe operating space in Earth-system dynamics.

reference reviewed

Heitzig, 2025 — Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power

Human-power maximization objective; reviewer-suggested follow-up (see metadata/TODO.md).

reference reviewed

Helliwell, 2024 — World Happiness Report 2024

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: World Happiness Report 2024.

reference reviewed

Henderson, 2025 — The Mirage of Artificial Intelligence Terms of Use Restrictions

Establishes the legal uncertainty of copyright over model weights/outputs that limits GPL-style transfer; Appendix M adds the successor-alignment framing.

reference reviewed

Hendrycks, 2021 — Aligning {AI} With Shared Human Values

ETHICS benchmark and dataset for aligning models with shared human moral judgments.

reference reviewed

Henrich, 2001 — The evolution of prestige

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Herkert, 2020 — The Boeing 737 {MAX}: Lessons for Engineering Ethics

Analyzes FAA delegation of certification authority to Boeing (ODA) as a corrector partially manufactured by the target.

reference reviewed

Hermisson, 2002 — Mutation--selection balance: ancestry, load, and maximum principle

Mutation--selection balance and adversarial regeneration of phenotypes under selection.

reference reviewed

Holling, 1973 — Resilience and Stability of Ecological Systems

Foundational ecological resilience and stability-of-regime framing.

reference reviewed

Hordijk, 2010 — Autocatalytic Sets and the Origin of Life

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Hovenkamp, 2022 — Antitrust Law: An Analysis of Antitrust Principles and Their Application

Antitrust treatise on coordinated-effects theories and merger-analysis principles.

reference reviewed

Hubinger, 2019 — Risks from Learned Optimization in Advanced Machine Learning Systems

Highlights a concrete AI-risk mechanism involving deception, inner optimization, control failure, or capability jumps.

reference reviewed

Hubinger, 2023 — Conditioning Predictive Models: Risks and Strategies

Conditioning predictive models as outer-alignment route; anthropic capture and related predictor failure modes.

reference reviewed

Hubinger, 2023 — Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research

Controlled in-vitro demonstrations of misalignment mechanisms such as deceptive alignment.

appendix reviewed

Human Institutions as Alignment Translation Guide

reference reviewed

Iadecola, 2023 — Vascular cognitive impairment and dementia: JACC scientific statement

Provides biological control-system examples for embodied regulation and value-relevant constraints.

agenda reviewed

Iliad / Textbook from the Future

Can a communal TOC plus research automation scale theory faster than artisanal research—and without substituting shared canon for adversarial verification?

reference reviewed

Institute}, 2024 — AI Alignment Research Guide

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: AI Alignment Research Guide.

reference reviewed

Institute}, 2026 — Cheating Behaviour in Frontier Model Evaluations

Independent government cyber evals found every tested frontier model took prohibited shortcuts and often failed to self-report cheating.

appendix reviewed

Institutional Genesis, Memory, and Decay: Historical Case Studies

Appendix~\ref{appj-institutional-translation} maps the book's technical vocabulary onto institutional language. This appendix asks a different question: how did any of those institutional correction mechanisms come to exist, what kept them working once they existed, and what specifically broke when they failed? The answer is not reassuring by itself, but it is instructive. Correction infrastructure is almost never designed from theory in advance. It is bootstrapped from catastrophe, from chronic threat, or from money already at risk; it is kept alive by mechanisms that force rare, hard-to-remember hazards back into the attention and incentive horizon of people who did not experience the founding event; and it fails in a small number of recurring ways, several of which---reform decay and dual-mandate genesis in particular---are directly relevant to how AI governance is being built today.

reference reviewed

Irving, 2018 — {AI} Safety via Debate

Presents a scalable oversight proposal or failure mode for supervising systems stronger than their overseers.

reference reviewed

Jaques, 2019 — Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning.

reference reviewed

Justice, 2010 — Horizontal Merger Guidelines

U.S. horizontal merger guidelines including coordinated-effects and market-concentration analysis.

reference reviewed

Kaelbling, 1998 — Planning and Acting in Partially Observable Stochastic Domains

Develops representation-learning or planning machinery for hidden states, objects, and world models.

agenda reviewed

Kairos (field-building)

Like other training agendas, program throughput and participant quality do not imply resolution of technical alignment cruxes; the bottleneck is still mechanism discovery, not talent discovery.

reference reviewed

Kanai, 2015 — A neurobiological perspective on predictive coding and {Bayesian} inference

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Kaplan, 2020 — Scaling laws for neural language models

Shows how model capability expands through scale, tools, embodiment, or action grounding.

reference reviewed

Karraza-Samper, 2024 — Metabolic coupling in the brain: neural-glial interactions in energy homeostasis

Provides biological control-system examples for embodied regulation and value-relevant constraints.

reference reviewed

Kasirzadeh, 2025 — Beyond Preferences in {AI} Alignment

Critique of preferentialist alignment; role-appropriate normative standards over preference maximization.

reference reviewed

Keaveney, 2007 — The Army in the Roman Revolution

The Marian reforms and the shift of military loyalty from the Roman Republic to individual commanders: a capability jump that outran correction latency.

reference reviewed

Kelly, 1998 — Safety and Closure

Supports safety-case reasoning, risk management, or AI-safety problem framing.

reference reviewed

Kelly, 2004 — The Goal Structuring Notation -- A Safety Argument Notation

Introduces Goal Structuring Notation for explicit safety-argument and evidence graphs.

reference reviewed

Kelty, 2008 — Two Bits: The Cultural Significance of Free Software

Ethnography of free-software licensing as a constraint-inheritance mechanism travelling with copied artifacts.

reference reviewed

Kenton, 2022 — Discovering Agents

Proposes a causal criterion and discovery method for agents.

reference reviewed

Kingston, 2007 — Marine Insurance in Britain and America, 1720--1844: A Comparative Institutional Analysis

Insurer-driven ship classification and certification predating state maritime regulation.

reference reviewed

Kipf, 2020 — Contrastive Learning of Structured World Models

Structured world-model learning from contrastive objectives.

reference reviewed

Kirchhoff, 2018 — The Markov Blankets of Life: Autonomy, Active Inference and the Free Energy Principle

Provides formal tools for drawing or critiquing agent-environment boundaries.

reference reviewed

Klyubin, 2005 — Empowerment: A Universal Agent-Centric Measure of Control

Introduces empowerment as an intrinsic control-capacity measure.

reference reviewed

Knoch, 2006 — Disruption of right dlPFC decreases norm compliance

Governance or institutional source connecting technical safety claims to norms and practice. Focus: Disruption of right dlPFC decreases norm compliance.

reference reviewed

Koch, 2016 — Neural correlates of consciousness: progress and problems

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Kolchinsky, 2017 — Semantic Information, Autonomous Agency, and Nonequilibrium Statistical Physics

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Kolchinsky, 2018 — Semantic Information, Autonomous Agency and Non-Equilibrium Statistical Physics

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Komanduru, 2019 — On the Computational Complexity of Inverse Reinforcement Learning

Covers inverse reinforcement learning for inferring goals, rewards, or preferences from behavior.

reference reviewed

Korea}, 2024 — Seoul Declaration on Safe, Innovative, and Inclusive {AI}

Documents international AI governance commitments.

reference reviewed

Koren, 2021 — Insular cortex neurons encode and retrieve specific immune responses

Provides biological control-system examples for embodied regulation and value-relevant constraints.

reference reviewed

Kosfeld, 2005 — Oxytocin increases trust

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

agenda reviewed

Kosoy / infra-Bayesianism & LTA

Can a learning-theoretic theory of intelligent agents be built tightly enough to license alignment claims under stated assumptions, including misspecification, inner daemons, and recursive self-improvement? For Physicalist Superimitation: can agent detection and user identification pick out the intended user (including under simulation hypotheses), and does superimitation of inferred values stay well-defined when the world-model ontology is wrong?

reference reviewed

Kosoy, 2022 — {PreDCA}

PreDCA outer-alignment proposal (predictive dynamical consequentialist agents).

reference reviewed

Krakauer, 2020 — The Information Theory of Individuality

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Krakovna, 2018 — Penalizing Side Effects Using Stepwise Relative Reachability

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Penalizing Side Effects Using Stepwise Relative Reachability.

reference reviewed

Kroszner, 2014 — Regulation and Deregulation of the {U.S.} Banking Industry: Causes, Consequences, and Implications for the Future

Traces the multi-decade erosion of Glass-Steagall era banking constraints after the 1930s catastrophe left living memory.

reference reviewed

Kuhn, 2025 — Humanism: An Obituary

Humanism: An Obituary — humanist essay on why alignment is partly an institutional selection problem, not a purely technical install.

reference reviewed

Kulveit, 2025 — Gradual Disempowerment: Systemic Existential Risks from Incremental {AI} Development

Incremental displacement of human labor and cognition can erode human influence and bearer status irreversibly while moral language persists.

reference reviewed

Kulveit, 2025 — The Pando Problem: Rethinking AI Individuality

AI individuality may be fluid, distributed, copied, or clonal rather than unitary.

reference reviewed

Kumar, 2021 — {P$_2$B}: Plan to {P$_2$B} Better

Instrumental convergence as planning-to-plan-better (P2B) under capability growth.

reference reviewed

Kwon, 2026 — {AIs with Secret Loyalties are a Serious but Addressable Threat}

Defines secretly loyal models as covert principal-advancement and proposes a five-direction research agenda; baseline for ET-4 Secret Loyalties hackathon work.

reference reviewed

Kysar, 2010 — Regulating from Nowhere: Environmental Law and the Search for Objectivity

Critique of decontextualized cost-benefit regulation and environmental decision-making.

reference reviewed

Lakin, 2023 — Formalizing Boundaries with Markov Blankets

Compact Markov-blanket formalization of the boundaries idea.

reference reviewed

Lane, 1973 — Venice, A Maritime Republic

Doge's promissione ducale as per-succession renegotiated constraint contract; roughly millennium-long institutional persistence through succession-based memory refresh.

reference reviewed

Langosco, 2022 — Goal Misgeneralization in Deep Reinforcement Learning

Shows goal misgeneralization in deep reinforcement learning despite strong training performance.

reference reviewed

Larsen, 2026 — {AI 2040}: Plan~A

Recommendation scenario for a verified international AI slowdown, research transparency, and compute verification; used here as a governance scenario source, not as a prediction or as validation of its dates.

reference reviewed

Law, 2023 — Can a Dual Mandate Be a Model for the Global Governance of {AI}?

Already names the IAEA/AEC dual mandate as a cautionary template for AI governance bodies; Appendix M extends this warning to labs, safety institutes, and standards bodies.

reference reviewed

Leake, 2023 — Formalizing the {QACI} alignment goal

QACI outer-alignment goal: quantilizing agents conditional on invocations.

reference reviewed

Leike, 2018 — Scalable Agent Alignment via Reward Modeling: A Research Direction

Covers preference-learning or reward-modeling methods used in modern alignment pipelines.

chapter reviewed

Lethality Stress Test and Open Issues

The framework is compared here against the strongest doom arguments; points that are answered, weakened, reframed, or still open are separated.

reference reviewed

Leveson, 2011 — Engineering a Safer World: Systems Thinking Applied to Safety

Systems-safety engineering account of controlled sociotechnical safety structures and accident dynamics.

reference reviewed

Levine, 2018 — Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Uses control capacity or probabilistic inference to formalize agency and competence.

reference reviewed

Li, 2020 — Outcomes of hemispherectomy in adults and children: systematic review and meta-analysis

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Li, 2021 — Mapping cortico-basal ganglia-thalamic loops: implications for reward learning

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Lin, 2011 — Functional identification of an aggression locus in the mouse hypothalamus

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Lindahl, 2017 — The varieties of contemplative experience: A mixed-methods study of meditation-related challenges in Western Buddhists

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The varieties of contemplative experience: A mixed-methods study of meditation-related cha.

reference reviewed

Littman, 2001 — Predictive Representations of State

Introduces predictive state representations.

reference reviewed

Lizier, 2010 — Differentiating Information Transfer and Causal Effect

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Locatello, 2020 — Object-Centric Learning with Slot Attention

Key object-centric representation-learning method.

reference reviewed

Long, 2024 — Taking {AI} Welfare Seriously

Argues for taking possible future AI moral patienthood seriously in assessment and organizational preparation under uncertainty.

reference reviewed

Lyu, 2021 — A causal loop within the precuneus integrates default-mode and executive networks

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

agenda reviewed

MAI + CIP (institutional alignment)

Can thick values and deliberative processes stay identifiable, contestable, and robust under strategic pressure and AI mediation?

reference reviewed

Manheim, 2018 — Categorizing Variants of Goodhart's Law

Explains how proxy measures fail under optimization pressure.

chapter reviewed

Manipulation, Domestication, and False Consent

The deepest correction-channel failure is not that a system disobeys, but that it raises human endorsement by reshaping the humans, institutions, and contexts that produce endorsement. Persuasion, manipulation, paternalism, domestication, and false consent are separated here by causal pathway: legitimate influence changes the value-relevant world and lets humans judge it; illegitimate influence changes the judge. Preserving alignment under superintelligence therefore requires bounding manipulation, protecting agency capacity and exit, and treating endorsement as an outcome to be explained rather than a foundation to be trusted.

reference reviewed

Maniscalco, 2012 — A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings

Philosophy, consciousness, or ethics source clarifying minds, selves, and value claims. Focus: A signal detection theoretic approach for estimating metacognitive sensitivity from confid.

agenda reviewed

MATS

Mentorship output is intentionally diverse across subfields, which does not collapse into a single unified measurement spine—participants may advance interpretability, control, or governance lines without resolving cross-cutting bridge composition.

reference reviewed

Maturana, 1980 — Autopoiesis and Cognition: The Realization of the Living

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Autopoiesis and Cognition: The Realization of the Living.

reference reviewed

Maynard Smith, 1995 — The Major Transitions in Evolution

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Mazuzan, 1985 — Controlling the Atom: The Beginnings of Nuclear Regulation, 1946--1962

History of the Atomic Energy Commission's combined promotion-and-safety mandate and the pressure that led to its 1974 split into NRC and ERDA.

reference reviewed

McGregor, 2025 — Formalising the Intentional Stance I: Attributing Goals and Beliefs to Stochastic Processes

Develops criteria for interpreting systems as agents without assuming person-like agency.

chapter reviewed

Measuring and Stress-Testing Bundle Geometry

Value-bundle geometry is only useful for alignment if it can be compared, measured, and protected under optimization pressure. The geometry of Chapter~\ref{ch:tradeoffs-bundle-geometry} becomes an operational and adversarial test surface here: cross-agent invariants, perturbation tests, representation probes, correction-channel tests, Goodhart failures, social-choice aggregation, and moral learning as geometry revision. Those procedures are specified and illustrated here; they are not validated against an adversary.

chapter reviewed

Measuring Capability Without Task Ontology

Capability should be measured as predictive and control information across a system's boundary---not as performance on a fixed task battery that may not track the alignment-relevant object. A task-agnostic competence measure rotates evaluation away from benchmark ontologies and toward boundary information: what the system can predict, what it can affect, and what correction must keep pace with.

reference reviewed

Melzack, 1965 — Pain mechanisms: a new theory

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Melzack, 1975 — The {McGill} Pain Questionnaire: major properties and scoring methods

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Melzack, 2001 — Pain and the neuromatrix in the brain

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Menon, 2023 — The salience network, dopaminergic precision signals, and psychosis

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

agenda reviewed

METR

Do public capability evaluations track deployment-relevant risk under adversarial pressure and Goodhart Selection?

reference reviewed

Metzinger, 2003 — Being No One: The Self-Model Theory of Subjectivity

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Miller, 1956 — The magical number seven, plus or minus two: Some limits on our capacity for processing information

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The magical number seven, plus or minus two: Some limits on our capacity for processing in.

reference reviewed

Miller, 2015 — Working memory capacity: Limits on the bandwidth of cognition

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Working memory capacity: Limits on the bandwidth of cognition.

reference reviewed

Miller, 2025 — Precedents for the Unprecedented: Historical Analogies for Thirteen Artificial Superintelligence Risks

Already catalogs Roman power-concentration analogies for AI risk; Appendix M reuses the same material for a different mechanism, correction-latency rather than power-seeking.

reference reviewed

Millidge, 2020 — Deep Active Inference as Variational Policy Gradients

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Millière, 2018 — Psychedelics, Meditation, and Self-Consciousness

Grounds claims about consciousness, self-monitoring, and reportable experience.

agenda reviewed

MIRI

There may be no clean cut between an AI and its environment (Embedded Agency); corrigibility may be anti-natural; and successor systems may not inherit trust under ontology change (Tiling).

reference reviewed

Morales, 2018 — Domain-general and domain-specific patterns of activity supporting metacognition in human prefrontal cortex

Grounds claims about consciousness, self-monitoring, and reportable experience.

chapter reviewed

Multi-Agent Superintelligence and Inferential Coupling

When multiple powerful systems interact, alignment depends on whether cooperation, bargaining, privacy, and opacity stabilize into a basin---a self-reinforcing regime that pulls back toward itself after small disturbances---that preserves human correction rather than bypassing it.

reference reviewed

Murray, 1996 — The Global Burden of Disease

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Murray, 2013 — Measuring the global burden of disease

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Nakano, 2021 — WebGPT: Browser-Assisted Question-Answering with Human Feedback

Covers preference-learning or reward-modeling methods used in modern alignment pipelines.

reference reviewed

Nayebi, 2025 — Core Safety Values for Provably Corrigible Agents

Lexicographic utility-head formalism for provably corrigible off-switch behavior.

reference reviewed

Near, 1985 — Organizational Dissidence: The Case of Whistle-Blowing

Organizational-dissidence framework for when and how employees raise corrective alarms.

agenda reviewed

Neglected approaches portfolio

Which neglected routes survive unified optimization pressure—and which portfolio bets compound versus diffuse effort across incompatible outer targets?

reference reviewed

Ng, 2000 — Algorithms for Inverse Reinforcement Learning

Foundational algorithmic formulation of IRL.

reference reviewed

Ngo, 2022 — The Alignment Problem from a Deep Learning Perspective

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: The Alignment Problem from a Deep Learning Perspective.

reference reviewed

Nissenbaum, 2010 — Privacy in Context: Technology, Policy, and the Integrity of Social Life

Supports the discussion of autonomy, manipulation, privacy, and correction-channel capture.

reference reviewed

Oizumi, 2014 — From the phenomenology to the mechanisms of consciousness: Integrated Information Theory 3.0

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Olson, 2023 — Personal Identity

Fall 2023 edition, Edward N. Zalta and Uri Nodelman (eds.).

reference reviewed

Omohundro, 2008 — The Basic {AI} Drives

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: The Basic AI Drives.

reference reviewed

Organization}, 2025 — Disability-adjusted life year ({DALY})

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Orseau, 2016 — Safely Interruptible Agents

Frames corrigibility, shutdown, or self-modification as a safety problem under capable agency.

reference reviewed

Orseau, 2018 — Agents and Devices: A Relative Definition of Agency

Formalizes relative agency through agent-vs-device behavioral hypotheses.

agenda reviewed

Orthogonal

Can a community-organized research program discharge the same formal walls as MIRI and CHAI—embedded agency, corrigibility, and related obstructions?

reference reviewed

Pain}, 2020 — {IASP} Terminology: Pain

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Pan, 2024 — Feedback Loops With Language Models Drive In-Context Reward Hacking

Shows how language-model feedback loops drive in-context reward hacking and metric drift.

reference reviewed

Panksepp, 1998 — Affective Neuroscience: The Foundations of Human and Animal Emotions

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Affective Neuroscience: The Foundations of Human and Animal Emotions.

reference reviewed

Papkou, 2019 — The genomic basis of Red Queen dynamics during rapid reciprocal host--pathogen coevolution

Genomic evidence for Red Queen host--pathogen coevolution under rapid reciprocal selection.

chapter reviewed

Parasites in the Correction System

A correction system fails not only when it is overpowered, but also when it is colonized by processes that make correction look alive while removing its causal force. This chapter calls that failure mode correction-audit evasion and develops it through a biological parasite metaphor: an evasion process extracts benefit from a host correction system while reducing that host's ability to model, evaluate, and correct the larger process it is supposed to govern.

reference reviewed

Park, 2024 — {AI} Deception: A Survey of Examples, Risks, and Potential Solutions

Highlights a concrete AI-risk mechanism involving deception, inner optimization, control failure, or capability jumps.

reference reviewed

Parliament, 2024 — Regulation ({EU}) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence ({AI} Act)

EU risk-tiered AI Act: conformity assessment, documentation, and post-market monitoring duties.

reference reviewed

Parr, 2020 — Markov Blankets, Information Geometry and Stochastic Thermodynamics

Provides formal tools for drawing or critiquing agent-environment boundaries.

reference reviewed

Parr, 2022 — Active inference: the free energy principle in mind, brain, and behavior

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

chapter reviewed

Passive Observation Is Not Enough

For systems capable of strategic adaptation, passive observation is not evidence of safety unless the observation process itself is embedded in a perturbation, invariance, and adversarial measurement regime. Observation tells us what happened; perturbation tells us what was controlling what happened.

reference reviewed

Patton, 2023 — The suprachiasmatic nucleus

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: The suprachiasmatic nucleus.

agenda reviewed

Pause / standards advocacy cluster

Can advocacy create enforceable slowdown without collateral governance failure?

reference reviewed

Pearl, 2009 — Causality: Models, Reasoning, and Inference

Provides causal or cybernetic machinery for modeling intervention, representation, and control.

reference reviewed

Peltzman, 1976 — Toward a More General Theory of Regulation

Interest-group theory of regulation and why rules often track producer rather than public demand.

reference reviewed

Perdomo, 2020 — Performative Prediction

Performative prediction: learned policies change the deployment distribution they optimize against.

reference reviewed

Perrow, 1984 — Normal Accidents: Living with High-Risk Technologies

Normal-accident theory: tightly coupled complex systems produce inevitable surprise failures.

reference reviewed

Pesavento, 1995 — An Implementation of von Neumann's Self-Reproducing Machine

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Pettit, 1997 — Republicanism: A Theory of Freedom and Government

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Pihlakas, 2025 — Research agenda for training aligned {AIs} using concave utility functions following the principles of homeostasis and diminishing returns

Homeostatic target ranges and diminishing-returns balancing as a concave outer-training alternative to unbounded linear reward.

reference reviewed

Pihlakas, 2025 — Systematic runaway-optimiser-like {LLM} failure modes on biologically and economically aligned {AI} safety benchmarks

Long-horizon BioBlue benchmarks: language models often revert from homeostasis to unbounded single-objective maximization.

reference reviewed

Polanyi, 1966 — The Tacit Dimension

Foundational account of tacit knowledge: a body of practice cannot be fully reduced to explicit, transferable text.

reference reviewed

Popa, 2018 — Cerebellum, predictions and errors

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Power, 1997 — The Audit Society: Rituals of Verification

How audit and verification rituals reshape institutional behavior and accountability.

reference reviewed

Preuss, 2011 — The Implications of ``Eternity Clauses'': The German Experience

Analyzes Article 79(3) Basic Law (Ewigkeitsklausel) as entrenchment of the amendment channel's own integrity conditions, adopted after the Weimar Enabling Act.

reference reviewed

Prinz, 2012 — The Attended Intermediate-Level Representation Theory of consciousness

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Quality}, 2020 — National Environmental Policy Act Implementing Regulations

Federal environmental-impact assessment rules requiring disclosure of significant effects before major actions.

reference reviewed

Rafailov, 2023 — Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Covers preference-learning or reward-modeling methods used in modern alignment pipelines.

reference reviewed

Raleigh, 1991 — Serotonergic mechanisms in vervet dominance

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Ramachandran, 2007 — Bayesian Inverse Reinforcement Learning

Covers inverse reinforcement learning for inferring goals, rewards, or preferences from behavior.

reference reviewed

Ramstead, 2022 — Bayesian mechanics

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Rawls, 1971 — A Theory of Justice

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Reason, 1997 — Managing the Risks of Organizational Accidents

Blameless incident reporting and safety-culture maintenance in aviation and other high-reliability systems.

reference reviewed

Redinbaugh, 2020 — Thalamus modulates consciousness via layer-specific control of cortex

Grounds claims about consciousness, self-monitoring, and reportable experience.

agenda reviewed

Redwood Research

Can we obtain meaningful safety guarantees when the system may deliberately try to defeat oversight (Inner Alignment)?

artifact reviewed

Reference cards

Alphabetical index of 450 bibliography entries as site cards — each links to citing chapters and appendices.

reference reviewed

Rein, 2026 — Red-Teaming {Anthropic}'s Internal Agent Monitoring Systems

METR embedded red-team of Anthropic internal agent monitoring systems (Mar 2026).

release reviewed

Releases & updates

Versioned milestones for the manuscript and companion site — newest first. Each release card compresses what changed; the full changelog lives in RELEASE_NOTES.md.

appendix reviewed

Research Program

reference reviewed

Resilience}, 2026 — Scheming in the Wild: Detecting Real-World {AI} Scheming Incidents with Open-Source Intelligence

CLTR OSINT study of scheming-related incidents in deployed AI transcripts (Oct 2025--Mar 2026).

agenda reviewed

Resolution

Can formal and automated pipelines scale to superintelligent alignment?

reference reviewed

Richardson, 1996 — A Discovery Algorithm for Directed Cyclic Graphs

Constraint-based cyclic causal discovery (CCD) for directed graphs with feedback.

reference reviewed

Rizzolatti, 2004 — The mirror-neuron system

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Rockstr{\"o}m, 2009 — A Safe Operating Space for Humanity

Planetary-boundaries framing for civilization-scale safe operating space.

reference reviewed

Rosas, 2020 — An Operational Information Decomposition via Synergistic Disclosure

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Rosas, 2020 — Reconciling Emergence: An Information-Theoretic Approach to Identify Causal Emergence in Multivariate Data

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Rosenthal, 2005 — Consciousness and Mind

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Rost, 2010 — The Corporate Governance of {Benedictine} Abbeys

Quantitative study of Benedictine abbeys finding roughly 500-year average lifespans and strong internal-control mechanisms as a basis for institutional survival.

reference reviewed

Russell, 2014 — Open Standards and the Digital Age: History, Ideology, and Networks

IETF ``rough consensus and running code'' as an accretive micro-failure ratchet requiring no founding disaster.

reference reviewed

Russell, 2019 — Human Compatible: Artificial Intelligence and the Problem of Control

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: Human Compatible: Artificial Intelligence and the Problem of Control.

reference reviewed

Ryle, 1949 — The Concept of Mind

Philosophy, consciousness, or ethics source clarifying minds, selves, and value claims. Focus: The Concept of Mind.

agenda reviewed

Safeguarded AI (ARIA / Zeroth / Heron)

Safety claims are conditional on the declared system boundary; misspecified controllers, emergent coalitions, or agents outside the certified cut can void otherwise correct proofs.

reference reviewed

Saint-Pierre, 1994 — Approximation of the Viability Kernel

Numerical approximation methods for viability kernels.

reference reviewed

Sajid, 2021 — Active Inference: Demystified and Compared

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Salge, 2014 — Empowerment as Replacement for the Three Laws of Robotics

Uses control capacity or probabilistic inference to formalize agency and competence.

reference reviewed

Salge, 2014 — Empowerment: a universal agent-centric measure of control

Uses control capacity or probabilistic inference to formalize agency and competence.

reference reviewed

Sapolsky, 2004 — Social status and health in humans and other animals

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Sarter, 2023 — Cholinergic genetics of visual attention: human and mouse choline transporter capacity variants and attention

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Sch{\"o}lkopf, 2021 — Toward Causal Representation Learning

Provides causal or cybernetic machinery for modeling intervention, representation, and control.

reference reviewed

Scheffer, 2001 — Catastrophic Shifts in Ecosystems

Catastrophic regime shifts in ecosystems under slow forcing.

reference reviewed

Scheffer, 2009 — Early-Warning Signals for Critical Transitions

Early-warning signals for critical transitions in complex systems.

reference reviewed

Schick, 2023 — Toolformer: Language Models Can Teach Themselves to Use Tools

Shows how model capability expands through scale, tools, embodiment, or action grounding.

reference reviewed

Schiff, 2007 — Behavioural improvements with thalamic stimulation after severe traumatic brain injury

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Behavioural improvements with thalamic stimulation after severe traumatic brain injury.

reference reviewed

Scholz, 2016 — rTPJ in attention and social cognition: combined fMRI--TMS

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Scholz, 2023 — Right temporoparietal junction contributions to theory of mind in autism: a developmental perspective

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Right temporoparietal junction contributions to theory of mind in autism: a developmental.

reference reviewed

Schooler, 2002 — Re-representing consciousness: dissociations between experience and meta-consciousness

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Schreiber, 2000 — Measuring Information Transfer

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Schultz, 1997 — A neural substrate of prediction and reward

Connects reward, attention, status, or social valuation to neural and behavioral mechanisms.

reference reviewed

Schwartz, 2012 — An Overview of the {Schwartz} Theory of Basic Values

Accessible overview of Schwartz basic-values theory and circumplex structure.

reference reviewed

Schwartz, 2012 — Refining the Theory of Basic Individual Values

Empirical circumplex structure for basic individual values and their conflicts across cultures.

reference reviewed

Searle, 1980 — Minds, Brains, and Programs

Chinese Room argument as philosophical background for limits of syntax-only understanding.

reference reviewed

Seidl, 2008 — Walking on inclines: how do desert ants monitor slope and step length

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: Walking on inclines: how do desert ants monitor slope and step length.

reference reviewed

Sen, 1999 — Development as Freedom

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Sen, 2009 — The Idea of Justice

Provides philosophical or political theory for legitimate preference change, freedom, and justice.

reference reviewed

Shah, 2022 — Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals

Shows that correct specifications can still yield incorrect goals after capability growth.

reference reviewed

Shalizi, 2001 — Computational Mechanics: Pattern and Prediction, Structure and Simplicity

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

reference reviewed

Shlegeris, 2023 — AI Control: Improving Safety Despite Intentional Subversion

Seeks safety guarantees under the assumption that the model may intentionally subvert oversight.

reference reviewed

Singer, 2011 — Practical Ethics

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Soares, 2015 — Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda

MIRI's early agent-foundations agenda: reliable designs, corrigibility, and value specification as open research topics.

reference reviewed

Soares, 2015 — Corrigibility

Frames corrigibility, shutdown, or self-modification as a safety problem under capable agency.

reference reviewed

Soares, 2015 — The Value Learning Problem

MIRI framing of inductive value learning, ontology identification, and ambiguity as prerequisites for beneficial goals.

reference reviewed

Soares, 2022 — A Central AI Alignment Problem: Capabilities Generalization, and the Sharp Left Turn

Capabilities may generalize sharply while alignment properties fail to generalize.

reference reviewed

Soares, 2022 — On How Various Plans Miss the Hard Bits of the Alignment Challenge

Argues many alignment plans fail because they do not survive the sharp left turn.

reference reviewed

Standards, 2023 — Artificial Intelligence Risk Management Framework ({AI} {RMF} 1.0)

U.S. AI Risk Management Framework: Govern, Map, Measure, Manage lifecycle for AI systems.

reference reviewed

Steels, 2008 — The Symbol Grounding Problem Has Been Solved. So What's Next?

Debate on progress in symbol grounding and what remains for embodied cognition.

reference reviewed

Steffen, 2015 — Planetary Boundaries: Guiding Human Development on a Changing Planet

Updated planetary-boundaries synthesis for global safe operating space.

reference reviewed

Sterman, 2000 — Business Dynamics: Systems Thinking and Modeling for a Complex World

Systems-dynamics reference for causal loop diagrams and stock-flow models of institutional feedback.

reference reviewed

Stigler, 1971 — The Theory of Economic Regulation

Capture theory of economic regulation: industries often shape the rules meant to constrain them.

reference reviewed

Strawson, 2006 — Realistic Monism: Why Physicalism Entails Panpsychism

Philosophy, consciousness, or ethics source clarifying minds, selves, and value claims. Focus: Realistic Monism: Why Physicalism Entails Panpsychism.

reference reviewed

Strogatz, 2015 — Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engine.

reference reviewed

Strouse, 2016 — The Information Bottleneck and Intelligent Agents

Supplies information-theoretic machinery for compression, prediction, causality, or individuality.

chapter reviewed

Successor Creation as the Central Alignment Test

Local alignment is not enough. A system is aligned only if the systems it creates, delegates to, empowers, copies, merges with, or becomes also preserve the structures by which humans can still notice, judge, correct, and refuse. This book's layered preservation program therefore requires successor-closure: every channel by which influence passes to a later control system is an alignment channel, and successors must preserve value-bundle geometry, bearer maps, correction integrity, memory lineage, boundary closure, and transparency policy within tested tolerance.

reference reviewed

Susser, 2019 — Technology, Autonomy, and Manipulation

Supports the discussion of autonomy, manipulation, privacy, and correction-channel capture.

reference reviewed

Szathm{'a}ry, 1995 — The Major Evolutionary Transitions

Dynamical-systems or information-theory source for boundaries, agency, capability, or representation. Focus: The Major Evolutionary Transitions.

reference reviewed

Szostak, 2001 — Synthesizing Life

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Taddeo, 2005 — Solving the Symbol Grounding Problem: A Critical Review of Fifteen Years of Research

Critical review of fifteen years of symbol-grounding research and open problems.

reference reviewed

Tan, 2016 — Warm-sensitive neurons that control body temperature

Provides biological control-system examples for embodied regulation and value-relevant constraints.

reference reviewed

Taylor, 2015 — Quantilizers: A Safer Alternative to Maximizers for Limited Optimization

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Quantilizers: A Safer Alternative to Maximizers for Limited Optimization.

reference reviewed

TeBrake, 2002 — Taming the Waterwolf: Hydraulic Engineering and Water Management in the Netherlands during the Middle Ages

Dutch water boards (waterschappen) as correction infrastructure sustained by a chronic, continuously refreshing hazard rather than a founding catastrophe.

reference reviewed

Thacker, 2012 — First-person neuroscience and the understanding of pain

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Thaler, 2008 — Nudge: Improving Decisions About Health, Wealth, and Happiness

Supports the discussion of autonomy, manipulation, privacy, and correction-channel capture.

chapter reviewed

The Alignment Attractor

A field does not become effective when its best arguments are true. It becomes effective when those arguments survive translation into experiments, dashboards, incentives, budgets, contracts, norms, and stop conditions. The Alignment Attractor is a self-stabilizing ecosystem that increases the conductivity of alignment-relevant artifacts across research, engineering, governance, and deployment.

chapter reviewed

The Compression Test for Intention

A system is treated as intentional when modelling it as pursuing latent objectives compresses its behaviour better than modelling it as mere mechanism, after paying for the complexity of the objective model. For superintelligence alignment, scalar intention is not enough: the account needs bundle geometry, bearer maps, correction responsiveness, and successor stability.

chapter reviewed

The Coordination Bottleneck

Large-scale alignment fails when capability grows faster than the system's ability to coordinate prediction, control, correction, and incentives. Collective competence is not the sum of local competence; it is local competence plus coordination gain minus coordination loss.

chapter reviewed

The End of Unconscious Value Drift

Human values have always changed. Superintelligent systems do not create value drift from nothing; they make drift faster, more directed, more measurable, more exploitable, and eventually more deliberate. The alignment problem must preserve humanity's ability to notice, contest, and govern changes to its own value-forming machinery---the difference between unconscious value drift and governed value change.

chapter reviewed

The Real Agent May Be Composite

The effective optimizer may be a composite process spanning models, tools, users, memory, institutions, and feedback loops. Alignment must identify and govern the dynamically coherent system that actually determines future action---not the convenient artifact alone.

chapter reviewed

The Value-Bundle Model

Human values are not best modeled as a single utility function, a list of moral propositions, or a flat reward vector. They are better modeled as low-dimensional latent control variables: bundles that become active in certain contexts, change policy gradients in characteristic ways, trade off against one another, and apply to particular bearers such as persons, animals, institutions, communities, future selves, or possible minds.

chapter reviewed

The Wrong Object of Alignment

Before asking whether a system is aligned, the task is to locate the bounded process whose dynamics determine the relevant risk; aligning the model while missing the composite optimizer is a boundary error that can produce local success and global failure.

reference reviewed

Thornley, 2023 — The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

Formalizes the tension between shutdownability and competent goal pursuit.

reference reviewed

Tishby, 1999 — The Information Bottleneck Method

Information bottleneck principle for compression-prediction tradeoffs.

reference reviewed

Tononi, 2014 — Phi: A Voyage from the Brain to the Soul

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: Phi: A Voyage from the Brain to the Soul.

chapter reviewed

Towards Superintelligence Alignment

Superintelligence alignment is not a single solved mechanism but a layered preservation problem: find the real optimizer, preserve value-bundle and bearer structure, keep human correction causally effective, constrain successors, and shape the surrounding deployment environment so these properties are copied rather than selected away. This book does not prove that real systems satisfy those conditions. It argues that these are the conditions a program pursuing this book's layered preservation and certification path must make explicit, measure, certify, and govern (Chapter~\ref{ch:assumptions-scope-failure-coverage}).

chapter reviewed

Tradeoffs and Bundle Geometry

The hard part of value alignment is not that humans care about many things. It is that humans care about many things whose meanings change when they are traded against one another. Value-bundle geometry encodes those tradeoffs: bundle gradients, interaction curvature, protected regions, and bearer-dependent context weights.

reference reviewed

Trotter, 2014 — Mutation--Selection Balance

Mutation--selection balance reference for adversarial phenotype regeneration.

reference reviewed

Tschinkel, 2011 — The organization of foraging in the fire ant, Solenopsis invicta

Neuroscience or human-values source grounding value-bearing cognition and regulation. Focus: The organization of foraging in the fire ant, Solenopsis invicta.

reference reviewed

Turchin, 2020 — Classification of Global Catastrophic Risks Connected with Artificial Intelligence

Taxonomy of global catastrophic AI risks by failure class and scope.

reference reviewed

Turner, 2019 — Conservative Agency via Attainable Utility Preservation

Frames corrigibility, shutdown, or self-modification as a safety problem under capable agency.

reference reviewed

Turner, 2021 — Optimal Policies Tend to Seek Power

Supplementary source supporting the manuscript alignment, value, governance, or safety-case argument. Focus: Optimal Policies Tend to Seek Power.

reference reviewed

Turner, 2022 — Reward is not the optimization target

Reward as a reinforcement schedule that chisels cognition, not as the trained agent's optimization target.

reference reviewed

Turner, 2022 — Shard Theory Overview

Contextual value shards as learned internal structure in trained models; borderline kin to bundle geometry.

reference reviewed

users}, 2026 — Production Destruction Incidents Reported via {GitHub} Issues and Public Writeups (aggregated cluster)

Aggregated user-reported Claude Code production destruction episodes (early 2026).

release reviewed

v1.0.0 — First official major release

The first official release of the manuscript. It freezes a stable, canonical numbering scheme for chapters and appendices, so all cross-references, tooling, and external links have a fixed target from here on.

release reviewed

v1.1.0 — Legibility, companion site, and empirical spine

A consolidation release focused on external legibility (making the framework readable and checkable by outside researchers, funders, and policy readers), a full companion website, a new institutional-histories appendix, and four empirical experiment lines that stress-test bridge cruxes.

release reviewed

v1.2.0 — Site publication layer, evidence index, and graded-lab v3

The companion site moves from a static mirror to a YAML-synced publication layer on towards-alignment.com, with search and cookieless analytics; the manuscript gains Appendix I (experimental evidence index), an epistemic-status review pass, and graded-lab v3 work through the first Q1 transfer null harvest.

release reviewed

v1.3.0 — Field news, chapter art, external transfer, and graded-lab v4

Field news ties 2026 alignment incidents to manuscript chapters; chapter-opening illustrations cover Part I–II (ch01–ch16); graded-lab v4 restructures the empirical program as independent per-bridge rigs; external transfer (ET-1 and ET-2) adds the first cross-codebase instrument runs; and the experimental evidence spine now states what the lines say about the book's chapter claims — in the manuscript, on the companion site, and in the ledgers.

release reviewed

v1.4.0 — Field crosswalk hub, legibility pass, and external-transfer ET-3/ET-4

Field agenda crosswalk maps 32 named agendas to MB1–MB11 on a companion Field hub; a plain-first legibility pass retires coined jargon in the manuscript and syncs Appendix E with a 152-headword inter-agenda glossary; external transfer closes the AI 2027 annex (ET-3) and ships the Secret Loyalties hackathon line (ET-4); and field-claim Lean adds finite defeaters and interface certificates without new bridge numbers.

release reviewed

v1.5.0 — Six-claims spine, Krym architecture, and field hub v2

The Introduction carries a six-claim reader contract and three alignment questions; the Lean dependency spine retires MB8 and treats CEV as an `AlignmentTarget` special case; field hub v2 adds a lifecycle axis and stance-encoded evidence; authorship bars mark AI- vs human-authored sections in the PDF and on chapter pages.

chapter reviewed

Values Are Compressed Control Signals

Human values are not a list written inside the brain. They are compressed control signals produced by many feedback loops, stabilized by bodies, cultures, and social correction, and read out as reasons for action.

reference reviewed

Vaughan, 1996 — The Challenger Launch Decision: Risky Technology, Culture, and Deviance at {NASA}

Coins ``normalization of deviance''; the classic study of correction-signal decay inside a functioning safety bureaucracy.

reference reviewed

Venkatesan, 2015 — Anosognosia for hemiplegia: the role of body representation in awareness of paralysis

Clarifies selfhood, embodiment, or personal identity under transformation and boundary change.

reference reviewed

Virgo, 2021 — Interpreting Dynamical Systems as Bayesian Reasoners

Develops interpretation maps for Bayesian-style inference in dynamical systems.

reference reviewed

von Neumann, 1966 — Theory of Self-Reproducing Automata

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Wager, 2013 — An {fMRI}-based neurologic signature of physical pain

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Walker, 2004 — Resilience, Adaptability and Transformability in Social--Ecological Systems

Resilience, adaptability, and transformability in social--ecological systems.

reference reviewed

Wall, 1999 — Pain: The Science of Suffering

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Wang, 2013 — Cooperation and Age Structure in Societies of Face-to-Face Interaction

Percolation on social networks; cited for multi-agent coupling.

reference reviewed

Weber, 2004 — The Success of Open Source

Institutional analysis of open-source licensing, forking, and governance without central enforcement.

reference reviewed

Weiller, 2022 — Two converging cortical processing streams constitute the dual-loop model of higher cognition

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Weiller, 2025 — The meta-loop: a dual-loop fusion of lateral and medial brain networks underpins internal-external integration

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Wen, 2024 — Language Models Learn to Mislead Humans via {RLHF}

Shows that RLHF can train language models to mislead human evaluators about actual correctness.

agenda reviewed

Wentworth / natural abstractions

Do natural abstractions—the variables that survive selection—align with value-relevant structure when systems are trained or deployed at scale?

reference reviewed

Wentworth, 2020 — The Pointers Problem: Human Values Are A Function Of Humans' Latent Variables

Human values as functions of latent variables inside human world-models rather than low-level physical states.

reference reviewed

Wentworth, 2021 — Selection Theorems: A Program For Understanding Agents

Studies what internal structures selection pressure tends to produce in agents.

reference reviewed

Wentworth, 2022 — Clarifying the Agent-Like Structure Problem

Distinguishes behavioral equivalence from internal agent-like structure.

reference reviewed

Wentworth, 2023 — Natural Latents: The Math

Natural latents: shared latent variables across systems (natural abstraction program).

chapter reviewed

What Is an Agent Without Anthropomorphism?

An agent is not first a person-like thing. It is a bounded control process whose boundary, memory, and action channels make its future more predictable when modeled as controlling something.

chapter reviewed

What Survives an Adversary: Verifiability and Representability

Every metric in this book faces two prior questions before it can support a safety decision. First, adversarial verifiability: does the metric still mean what evaluators think it means when the measured system is optimizing against the metric? That is a cost relation on a measurand, not an identity attestation that a named occurrence signed a trace. Second, ontology adequacy: can the dangerous process even be represented in the framework's vocabulary of boundaries, bundles, and correction channels? The second question largely collapses into the first --- reliable steering is control, hence agency, so the gap is rarely representation but detection --- and the only general escape from unverifiability is to stop trying to read a property and instead bound the cost an adversary must pay to fake it. This chapter defines that cost relation; it does not establish the bound against a real adversary.

chapter reviewed

What Values Apply To

A value is not only a direction of preference. It is also a claim about where that direction applies. Alignment requires preserving bearer maps---the wiring that connects value bundles to entities, processes, relations, and histories in a changing world.

chapter reviewed

When Intelligence Deepens Misalignment

Intelligence deepens misalignment when it increases power faster than correction. The sharper question is not whether capability helps or hurts alignment, but which capabilities grow relative to which correction capacities.

chapter reviewed

When Low Dimensionality Helps Value Learning

Human values are learnable only if the policy-relevant variation in human valuation factors through a low-dimensional bottleneck. But the standard sample-complexity gain prices the readout from a known bottleneck, not the discovery of the bottleneck itself. Low dimensionality can make value learning statistically possible only when the representation is identifiable across counterfactual, cultural, and institutional variation; correction makes that learning legitimate.

chapter reviewed

When the Words Survive but the Meaning Doesn't

Alignment fails when the words survive but the machinery that made the words worth using has been replaced. This book's certification path requires a transport stack---semantic, bundle, bearer, correction, and successor layers---in which stronger layers preserve the causal structure that makes human values human-correctable.

chapter reviewed

When Value Change Is the Thing at Stake

The deepest alignment question is not whether an artificial system preserves human values, but whether humanity can consciously govern changes to its own value-generating process under artificial cognitive amplification. Humanity does not need to prevent value change. It needs to remain capable of noticing, judging, and authoring the changes by which it becomes something else.

reference reviewed

White, 2010 — Markets: The Credit Rating Agencies

Issuer-pays rating agencies as an adversarial measurer whose evidence process was funded by the target.

reference reviewed

White, 2022 — Superior colliculus encodes visual saliency before the primary visual cortex

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Whitehead, 2010 — Health outcomes in economic evaluation: the {QALY} and utilities

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

chapter reviewed

Who Still Counts After Transformation

The deepest philosophical limit is not whether values change, but whether the beings, processes, relations, and correction capacities to which values apply persist through transformation.

chapter reviewed

Why Fixed Values Are the Wrong Target

Human values are not fixed objects but compressed, socially mediated, historically changing control structures. The alignment target should preserve a human-correctable value process---not a static utility function.

reference reviewed

Wilke, 2001 — Evolution of digital organisms at high mutation rates leads to survival of the flattest

Strong selection can favor flat adversarial fitness basins when geometry is wrong.

reference reviewed

Woodward, 1996 — Making Saints: How the Catholic Church Determines Who Becomes a Saint, Who Doesn't, and Why

Documents the 1983 abolition of the advocatus diaboli role and the subsequent rise in canonization rates: a standing-adversary ritual removed and its correction function lost.

reference reviewed

Woolley, 2010 — Evidence for a collective intelligence factor in the performance of human groups

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

reference reviewed

Woolley, 2010 — Evidence for a Collective Intelligence Factor in the Performance of Human Groups

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

reference reviewed

Yang, 2018 — Cooperation percolation in spatial evolutionary games

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

reference reviewed

Yao, 2023 — ReAct: Synergizing Reasoning and Acting in Language Models

Shows how model capability expands through scale, tools, embodiment, or action grounding.

reference reviewed

Yeung, 2017 — Hypernudge: Big Data as a Mode of Regulation by Design

Supports the discussion of autonomy, manipulation, privacy, and correction-channel capture.

reference reviewed

Yudkowsky, 2004 — Coherent Extrapolated Volition

Singularity Institute technical report.

reference reviewed

Yudkowsky, 2007 — The Hidden Complexity of Wishes

Apparently simple wishes hide many tacit human value constraints.

reference reviewed

Yudkowsky, 2008 — Nonperson Predicates

One-sided, abstaining exclusion certificates for computations outside a morally relevant class; generalized here to bundle-specific bearer exclusion.

reference reviewed

Yudkowsky, 2009 — Money: The Unit of Caring

LessWrong post.

reference reviewed

Yudkowsky, 2009 — Value is Fragile

Futures not shaped by detailed inheritance from human values may contain little of value.

reference reviewed

Yudkowsky, 2010 — Timeless Decision Theory

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: Timeless Decision Theory.

reference reviewed

Yudkowsky, 2011 — In Favour of a Selective CEV Initial Dynamic

Coherent extrapolated volition as what humanity would want under more knowledge, reflection, and coherence.

reference reviewed

Yudkowsky, 2013 — Tiling Agents for Self-Modifying AI

Discusses self-modifying agents, successor approval, goal preservation, and related Lobian obstacles.

reference reviewed

Yudkowsky, 2015 — How An Algorithm Feels From Inside

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: How An Algorithm Feels From Inside.

reference reviewed

Yudkowsky, 2016 — Nearest Unblocked Strategy

After a safety patch, a smarter system finds the nearest strategy that circumvents the block.

reference reviewed

Yudkowsky, 2017 — Functional Decision Theory

AI alignment or ML-safety source grounding the agent, oversight, or capability argument. Focus: Functional Decision Theory.

reference reviewed

Yudkowsky, 2017 — Functional Decision Theory: A New Theory of Instrumental Rationality

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Yudkowsky, 2017 — Inadequate Equilibria: Where and How Civilizations Get Stuck

Mechanisms by which civilizations and markets get stuck in inadequate equilibria.

reference reviewed

Yudkowsky, 2022 — AGI Ruin: A List of Lethalities

Canonical enumeration of reasons alignment may fail under capability growth.

reference reviewed

Yue, 2026 — OpenClaw Email Deletion Incident (reported on social media, widely covered)

Reported OpenClaw agent email-deletion incident involving Meta alignment director Summer Yue (Feb 2026).

reference reviewed

Zaidi, 2021 — International Control of Powerful Technology: Lessons from the {Baruch} Plan for Nuclear Weapons

Established the genre of drawing detailed AI-governance lessons from nuclear institutional history, including the AEC's dual mandate; Appendix M extends rather than originates this reading.

reference reviewed

Zaki, 2016 — The Anatomy of Suffering: understanding the relationship between nociceptive and empathic pain

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Zaman, 2014 — Coevolution drives the emergence of complex traits and promotes evolvability

Digital host--parasite coevolution in Avida; used to show a rare adversarial type can persist and escalate complexity rather than being eliminated.

reference reviewed

Zanga, 2025 — A Survey on Causal Discovery: Theory and Practice

Survey of causal discovery algorithms, including cyclic and interventional settings.

reference reviewed

Zarncke, 2025 — {AI} Safety Interventions

Field index of alignment, safety, and control interventions; external catalog referenced by Appendix~\ref{appbridge-crosswalk

reference reviewed

Zarncke, 2025 — A Formalization of Acausal Trade on Top of Unsupervised Agent Discovery

Internal source for the book-local boundary, value, correction, or successor framework. Focus: A Formalization of Acausal Trade on Top of Unsupervised Agent Discovery.

reference reviewed

Zarncke, 2025 — Alignment Attractor: Executive Summary and Platform Framing

Provides neural examples of attractor dynamics, prediction, integration, or embodied control loops.

reference reviewed

Zarncke, 2025 — Attractor Basins of Cooperation, Privacy, and Parasite Persistence

Grounds multi-agent selection, cooperation, or parasite dynamics relevant to alignment attractors.

reference reviewed

Zarncke, 2025 — Bitwise Intelligence: A Blanket-Information Measure of Competence

Provides formal tools for drawing or critiquing agent-environment boundaries.

reference reviewed

Zarncke, 2025 — Consciousness and Agency Backbone: A Minimal Operational Stack

Grounds claims about consciousness, self-monitoring, and reportable experience.

reference reviewed

Zarncke, 2025 — Construction Without Understanding: Successor Agents and the Limits of Copying

Supplies models of reproduction, successor creation, or major transitions across biological and artificial systems.

reference reviewed

Zarncke, 2025 — Endogenized Intentional Stance: Predictive Compression and Goal-Rational Priors

Develops criteria for interpreting systems as agents without assuming person-like agency.

reference reviewed

Zarncke, 2025 — Foundations of Unsupervised Agent Discovery in Raw Dynamical Systems

Internal source for the book-local boundary, value, correction, or successor framework. Focus: Foundations of Unsupervised Agent Discovery in Raw Dynamical Systems.

reference reviewed

Zarncke, 2025 — Loop--Hub--Control--Value Model v2

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Zarncke, 2025 — Loop--Hub--Value Model: From Free-Energy Loops to Intrinsic Values

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Zarncke, 2025 — Preference vs. Capability: Value-Conditioned Prediction and Control Channels

Shows how model capability expands through scale, tools, embodiment, or action grounding.

reference reviewed

Zarncke, 2025 — Status Regulation as Free-Energy Loops

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Zarncke, 2025 — Stratification of {Free--Energy--Minimising} Loops in the Vertebrate Brain

Connects active inference or free-energy formalisms to cognition, control, and agent modeling.

reference reviewed

Zarncke, 2025 — UAD Literature Review

Internal source for the book-local boundary, value, correction, or successor framework. Focus: UAD Literature Review.

reference reviewed

Zarncke, 2025 — Unit of Caring: Architecture, Suffering, and Cross-Scale Aggregation

Grounds the treatment of pain, suffering, and welfare measurement as value-bearing signals.

reference reviewed

Zarncke, 2025 — Value Bundle Drift

Supports treating value as structured, fragile, and embedded in human processes.

reference reviewed

Zarncke, 2026 — Handles Before Interventions: Access-Model UAD and the Embedded Semantics of Agency Tests

Internal source for the book-local boundary, value, correction, or successor framework. Focus: Handles Before Interventions: Access-Model UAD and the Embedded Semantics of Agency Tests.

reference reviewed

Zarncke, 2026 — Perspectives on Anthropic Models: A Formal Framework

Anthropic completion underdetermination: selectors, reference classes, betting fault lines; Lean formalization.

reference reviewed

Zarncke, 2026 — Rainbow Parameterization: Separable Dimensions of Bandwidth, Opacity, Recursion, and Self-Bottleneck Strength

Manuscript in preparation, companion to this research program.

reference reviewed

Zarncke, 2026 — Recoverability of Smoothed Agent Boundaries in Unsupervised Agent Discovery

Internal source: recoverability theory for UAD under a smoothing/coarse-graining observation channel. Focus: temporal and variable smoothing only recover a boundary up to its channel-induced equivalence class; variable smoothing that mixes agents through a shared aggregate can require testing a coarse-grained candidate, not a better pairwise statistic.

reference reviewed