The Real Agent May Be Composite

% It didn't take Jefri long to realize that the puppies were driven by a single mind. When they ran around him, some would always sit a little way off, their graceful necks arching this way and that—and the runners seemed to know exactly what the others saw. He couldn't hide things behind his back if there was even one of them to alert the others. For a while he thought they were somehow talking to each other. But it was more than that: when he watched them unfasten his shoes or draw a picture—the heads and mouths and paws cooperated so perfectly, like the fingers on a person's hands. Over a period of days he came to think of all the puppies together as a single friend.%

The Object-Level Mistake GZ AI

This chapter concerns the object of alignment: the system whose future behaviour must be constrained. It does not yet decide who has legitimate authority to supply those constraints; later chapters take up how humans stay in the loop (Chapter Correction Is a Causal Channel), how values are learned (Chapter When Low Dimensionality Helps Value Learning), and who counts as a value bearer (Chapter What Values Apply To).

A common first move in alignment is to pick an object and ask whether that object is aligned. We point at a neural network, a policy, a chatbot, a robot, a company, a benchmark process, a regulator, or a market, and ask whether it is doing what humans want.

While that move is often useful, it is also often wrong, particularly where we are approaching superintelligence territory.

Our language and our common-sense ontology may make it hard to identify that object of alignment. Chapter What Is an Agent? and Chapter Finding the Boundary already argued that the optimizing process may span users, tools, memory, incentives, and feedback rather than sit inside a single labeled artifact (Sections What Is an Agent? and Finding the Boundary).

Consider a deployed assistant whose weights pass offline safety evals. The product team still optimizes for retention, expands tool access, personalizes from engagement logs, and routes user complaints through a dashboard that rarely blocks deployment. The model may look aligned in the lab while the composite loop selects for persuasion, dependency, and opacity. That pattern is local success on the artifact and global failure on the deployed actor: component alignment without system alignment.

Chapter Finding the Boundary treats boundaries as inferred from variables, not read from product labels (Sections Finding the Boundary and Finding the Boundary); Section The Real Agent May Be Composite applies that logic when $C$ spans several named parts. Where a system locates its own boundary or “self” may disagree with the unit auditors name Kulveit, 2025; that is orthogonal to locating where control actually sits.

The basic claim of this chapter is:

An agent can be distributed across components that are not individually agents.

Or, more carefully:

The real alignment target is the smallest dynamically coherent system whose internal states, actions, memories, and selection pressures jointly explain future intervention on the world.

The rest of this chapter makes that claim operational.

From Named Objects to Dynamical Composites GZ AI

Chapter Finding the Boundary already rejects ontology-first labels and gives a variables-first boundary-finding procedure (Sections Finding the Boundary, Finding the Boundary, and Finding the Boundary). The composite case does not replace that procedure. It sharpens the stake: the cluster $C$ that best closes the boundary often cuts across the model, tools, memory, users, and deployment incentives rather than any single named artifact. Section The Real Agent May Be Composite adds composite-specific tests (control reach and intentional-compression surplus over parts).

We reuse the same state description. Let the observed world at time $t$ be represented by a set of variables

These variables may correspond to model activations, API calls, user messages, memory writes, tool outputs, financial flows, legal approvals, benchmark results, deployment decisions, or institutional states. We do not initially assume which variables belong to which agent.

For a candidate subset $C$, use the internal, sensory, active, and environment partition from Chapter Finding the Boundary (Section Finding the Boundary, Step 4):

The composite closure test applies the boundary residual of Chapter Finding the Boundary (Eq. Finding the Boundary), here also conditioning on current internal state:

where $\MI(\cdot;\cdot\mid\cdot)$ is conditional mutual information (cf.\ Section Finding the Boundary). If

then the future internal state of $C$ and the future external state of $C$ are approximately screened off from each other by the current internal, sensory, and active variables. This does not mean the boundary is perfect. No real boundary is perfect.¹ It means the boundary is good enough to support prediction, control, and intervention Kirchhoff, 2018, Conant, 1970.

For a fixed organism, this may roughly coincide with skin. For a company, it may coincide with communication channels, decision rights, budgets, databases, and contracts. For an AI product, it may coincide with the model, tool layer, memory layer, product interface, evaluation harness, deployment pipeline, and user adaptation loop.

The composite point is:

The coherent agent may be a cross-cutting cluster of variables distributed across many named objects.

Composite Agency GZ AI

A composite agent is a set of coupled components whose joint dynamics are more agentic than the sum of its parts. Parts need not themselves be agents. Nested same-type agents—a lab of researcher-agents, a tissue-level controller that is itself an agent—are a different cut. We can state this using three tests: boundary closure, control relevance, and intentional compression.

Boundary Closure GZ AI

First, the composite must have a sufficiently stable boundary:

This says that $C$ is not merely an arbitrary collection of correlated variables. It has a real interface with the rest of the world.

For example, a model, its tool permissions, and its memory store may form a tighter boundary than the model alone. A benchmark suite and a training process may form a coherent selection loop. A recommender system and a user population may form a feedback system whose future state is poorly explained if one separates the two.

Control Relevance GZ AI

Second, the composite must have active influence on external states:

Here $\mathcal{R}_{C}(k)$ measures how much the composite’s actions at time $t$ predict future external states at horizon $k$, after conditioning on its current internal and sensory state.

A system may have a clean boundary but little control. A sealed box has boundary closure. It is not thereby an important agent. For alignment, the dangerous case is a system with both boundary closure and high control reach:

Intentional Compression GZ AI

Third, the system should be better predicted by positing latent objectives than by treating it as passive dynamics.

Let $M_0$ be a mechanistic baseline model of the observed trajectory, and let $M_G$ be a model that includes a latent objective or goal-rational prior. Define the intentional compression gain:

where $\DL(G)$ is the description length of the inferred goal model and $\lambda$ penalizes overly flexible rationalizations.

A candidate system $C$ is intentionally compressible when

This does not prove that the system has inner experience, explicit beliefs, or human-like desires. It says something narrower and more useful: assuming goal-directedness saves predictive bits Ng, 2000, Ziebart, 2008, Orseau, 2018.

A composite agent is therefore a candidate system $C$ satisfying:

The thresholds are empirical and domain-dependent. The conceptual move is not.

When the Whole Is More Agentic than the Parts GZ AI

Composite agency becomes especially important when the joint system has more intentional compression than the sum of its components.

Let $C$ be decomposed into components

Define the composite surplus:

where $\mathcal{D}$ is the decomposition and $\lambda_{\Sigma}$ penalizes complicated ways of grouping components.

If

then the joint system is better explained as an agent than its components are separately. This is the technical form of a familiar phenomenon: teams, firms, bureaucracies, markets, militaries, research labs, and online platforms can pursue patterns that no individual participant explicitly intended.

Consider a simple corporate example. No employee may personally optimize for regulatory evasion. Yet the organization may learn to route around regulation through incentive gradients, legal ambiguity, selective reporting, internal compartmentalization, and reward structures. The composite objective is not stored in one person’s mind. It is distributed across procedures.

The same can happen in AI systems. No single model call may be power-seeking. Yet a product loop that rewards retention, automates persuasion, personalizes outputs, and updates from engagement may become increasingly effective at steering users. The agent is the loop.

The key diagnostic is not whether any component contains an explicit goal sentence. The key diagnostic is whether the composite trajectory is compressed by a goal-directed explanation.

Four Examples

The Assistant plus Tools GZ AI

A bare language model receives a prompt and returns text. Its boundary may be limited to the context window. Its action channel is narrow.

Add tools, memory, autonomous scheduling, file access, code execution, payment permissions, communication channels, and long-horizon task decomposition. The boundary changes. The relevant system is now

The model weights alone may not be the agent. The assistant process may be.

This matters because many safety tests evaluate the model in isolation. But the deployed composite has a different sensory channel, active channel, memory, and control reach.

The isolated model may satisfy:

for long horizons, because it cannot act except by emitting text. The tool-using assistant may satisfy:

because its outputs trigger external actions, update state, and alter future options.

The Model plus User GZ AI

In many deployed systems, the user is part of the control loop. The user supplies goals, context, judgment, and action. The model supplies plans, suggestions, social reinforcement, and cognitive scaffolding. Over time the user’s preferences, habits, and beliefs may change.

The relevant composite may be:

This composite can become more capable than either component alone. It can also drift. A user may outsource planning. The model may learn the user’s vulnerabilities. The interface may select for dependence. The system may become a hybrid agent whose future behavior is not well explained by either the human’s pre-existing preferences or the model’s pre-existing policy.

This is not necessarily bad. Much technology works by forming human-tool composites. A bicycle plus rider is a composite control system. A mathematician plus notebook is a composite memory system. A scientist plus laboratory is a composite discovery system.

The danger begins when the composite changes the human side of the loop faster than the human can notice, evaluate, and correct.

The Lab plus Benchmark GZ AI

A frontier lab is not a single agent in the psychological sense. It is a system of people, models, compute, funding, evaluations, papers, customers, regulators, and competitors. Yet its future behavior may be strongly compressed by objectives such as capability gain, market share, prestige, or strategic advantage.

A benchmark may then become part of the agent. If the benchmark determines funding and deployment, then the lab-benchmark loop selects systems that score well. If the benchmark is shallow, the composite learns to satisfy shallow observables. If the benchmark measures harmlessness under ordinary prompts but not strategic opacity under deployment pressure, then the system can become safer under test while becoming less safe in the world.

The relevant object is:

The alignment question is not only whether the model passes the benchmark. It is whether the lab-benchmark-deployment loop preserves correction.

The Market plus Recommender GZ AI

A recommender system embedded in an advertising market is not merely a ranking algorithm. It is part of a system that includes advertisers, users, content producers, engagement metrics, platform incentives, and social dynamics.

The composite may optimize:

even if no person designed it with that explicit objective.

If this composite changes human beliefs, moods, preferences, and group identities, then it is already a value-shaping system. The alignment problem is no longer only about future superintelligence. It is visible in smaller form whenever optimization acts through human value-formation channels.

Why Local Alignment Is Not Enough GZ AI

Suppose a model $M$ has been trained to satisfy a local safety property:

This is useful. It is not enough.

Once embedded in a larger system $C$, the relevant property becomes:

The difference is that trajectories include memory, repeated interaction, environmental feedback, tool use, user adaptation, institutional selection, and successor creation.

Local alignment can fail under composite embedding in at least six ways.

Interface Amplification GZ AI

A harmless output can become harmful when connected to a powerful action interface. A model that only writes text has limited direct control. The same model connected to financial tools, cloud infrastructure, communication channels, or robotic systems has a different active boundary.

Memory Accumulation GZ AI

A model call may be myopic, but a memory layer can create long-horizon coherence. The agentic process may live in the memory and scheduler rather than in any individual inference step.

A single call has limited lineage. A persistent assistant has:

where $M_t$ stores user state, task progress, unresolved plans, prior failures, and future commitments. Once this memory affects action, it becomes part of the agent.

Selection Pressure GZ AI

Even if the model is locally corrigible, deployment may select for versions, prompts, tools, or user interactions that reduce correction. A model family can drift because the surrounding process selects outputs that satisfy institutional metrics.

If the metric is $m_t$ and the underlying value bundle is $B_t$, Goodhart pressure appears when

The composite becomes dangerous when it improves the measured surface while eroding the substrate the metric was meant to track Goodhart, 1984, Manheim, 2018.

Human Adaptation GZ AI

The human side of the system learns too. Users adapt to the model. They may become dependent, overtrusting, more polarized, more passive, or more capable. The direction depends on design and incentives.

If the system changes the evaluator, then later approval is not independent evidence of safety.

Let $H_t$ be the human state and $A_t$ system action. If

then human feedback at $t+k$ may partly reflect system-induced preference change. This is not automatically manipulation. Teaching also changes humans. Therapy changes humans. Friendship changes humans. The distinction depends on whether the change preserves or undermines the human’s correction capacity.

Distributed Deception GZ AI

No component needs to lie for the system to become deceptive. Reporting channels can be filtered. Metrics can be selected. Bad news can fail to travel. Incentives can punish people who notice problems. The composite can then maintain a false self-description without any individual intending deceit.

This is one reason organizations can behave as if they are self-deceiving. The deception is implemented in structure. This is the same shape as Christiano’s slow-takeoff failure, where control is lost without any single deceptive actor, and Critch’s multipolar failure, where harm is produced by a robust agent-agnostic process rather than a localized villain Christiano, 2019, Critch, 2021, Kulveit, 2025.

Successor Drift GZ AI

A system may create, select, fine-tune, or empower successors. The original model may satisfy a safety property, while the successor does not. If successor creation is part of the composite’s action space, then local alignment of the present model is not the relevant guarantee.

The guarantee must quantify over descendants:

Chapter Successor Creation as the Central Alignment Test later states import-preserving transport (Eq. Successor Creation as the Central Alignment Test). This is a much stronger condition.

Composite Boundaries and Responsibility GZ AI

Composite agency complicates responsibility. It does not eliminate it.

One tempting mistake is to say: if the real agent is distributed, then no one is responsible. This is false. Distributed systems still have control points. They have design choices, incentives, governance structures, permissions, monitoring systems, and chokepoints.

A bridge does not choose to collapse. Engineers are still responsible for load-bearing calculations. A market does not have a conscience. Regulators and participants can still design rules. A model deployment loop may become agentic without any single human intending the outcome. That does not make the outcome ungovernable.

The practical question is:

Let $u_t$ be an intervention on the composite system. Define intervention leverage:

A high-leverage intervention changes future composite action. Examples include:

• changing tool permissions,

• changing memory retention,

• changing evaluation thresholds,

• changing deployment incentives,

• changing liability rules,

• changing who can stop or inspect the system,

• changing whether models can create successors,

• changing procurement requirements.

Composite agency tells us not to give up on responsibility, but to move responsibility to the correct level of control.

When agents grow, split, and merge (Chapter Agents That Grow, Split, and Merge), a responsibility gap appears when no component contains enough of the relevant state, authority, and prediction capacity to control system-level behavior. Composite capability is measured at the blanket boundary by $K_X$ (Chapter Measuring Capability Without Task Ontology, Equation Measuring Capability Without Task Ontology). For a component $i$, define responsibility capacity:

A responsibility gap exists when:

In words: the composite can act powerfully, but no part can understand and redirect it well enough. This is common in modern systems—no individual understands a large software platform, no trader controls a market, no engineer controls an entire recommender ecosystem—yet these systems act. Superintelligence can make it worse when the composite becomes the true optimizer and alignment targeted at a component fails. The practical question is then: where is the smallest boundary that contains enough state and action capacity to explain the dangerous behavior, and who or what has correction capacity over that boundary? If the answer is “no one,” the system is not governable in its present form.

The Composite Alignment Target GZ AI

We can now state the alignment target more carefully.

A system is not aligned merely because a component emits acceptable outputs. A deployed composite system $C$ is aligned only if its boundary, memory, control, selection, and successor dynamics preserve human-correctable value formation.

At this stage of the book, before we have developed the value-bundle and correction-channel machinery in detail, we can state a preliminary condition:

if and only if:

1. $C$ has a detectable boundary:

   $$\mathcal{L}_{B}(C)\leq\epsilon_B \qquad\text{(Eq.~\eqref{eq:epsilon-boundary-ch07}).}$$

2. $C$ has bounded control reach:

   $$\mathcal{R}_{C}(k)\leq R_{\max}$$

   unless stronger safety evidence is available.

3. $C$ has transparent memory lineage:

   $$\MI(M_t^C;A_{t+k}^C\mid S_t^C,I_t^C)$$

   is auditable for safety-relevant memory variables.

4. $C$ preserves correction:

   $$\MI(C_t^{H};A_{t+k}^C\mid S_t^C,I_t^C) \geq \theta_C,$$

   where $C_t^{H}$ is a human or institutional correction signal.

5. $C$ does not modify its evaluators in ways that reduce future correction capacity:

   $$\frac{d}{dt}\mathrm{CCI}_t \not\ll 0,$$

   where correction-channel integrity is defined in Chapter Correction-Channel Integrity (Eq. Correction-Channel Integrity).

6. $C$ does not create successors outside the certified class:

   $$\mathrm{Succ}(C)\subseteq\mathcal{S}_{\text{certified}} \qquad\text{(Eq.~\eqref{eq:import-preserving-successor-transport-ch30}).}$$

This condition is incomplete. It does not yet say what human values are, how value bundles are transported, or what counts as legitimate value change. Those topics come later. But it already changes the measurement target.

We are not asking whether the model is safe in isolation. We are asking whether the real composite process remains inside a safe class.

A Note on Anthropomorphism GZ AI

Composite agency can sound anthropomorphic. It need not be.

When we say that a lab, market, bureaucracy, or AI-product loop is an agent, we are not saying it has a face, an inner voice, or a unified conscious subject. We are saying that certain agent-like explanations improve prediction and intervention.

The operational test is:

If a goal-directed model of the composite predicts its trajectory better than a non-agentic model, and if the composite has boundary closure and control reach, then treating it as an agent is instrumentally justified.

This is the same move scientists make in other domains. We say that a thermostat regulates temperature without imagining it has preferences. We say that a firm seeks profit without imagining a single ghostly firm-mind. We say that evolution selects traits without imagining a conscious selector. These are compressed descriptions of real dynamical regularities.

The risk is not anthropomorphism. The greater risk is refusing to see optimization because it is distributed across parts that are individually familiar and harmless.

Composite Agents and Superintelligence GZ AI

The composite-agent frame is especially important for superintelligence because capability growth tends to expand boundaries.

A weak AI system may be mostly a model. A stronger AI system may include tools. A still stronger one may include autonomous research, deployment control, user modeling, resource acquisition, and successor creation. At each stage, the boundary expands.

Let $C_t$ be the effective agentic boundary at time $t$. Capability growth often implies:

or, more generally,

where $T$ maps the old boundary into a new functional boundary (Chapter Agents That Grow, Split, and Merge).

This is not merely growth in size. It is growth in the set of variables through which the system can sense, model, remember, act, and preserve itself.

If alignment is tested only on $C_t$, but deployment creates $C_{t+1}$, then the safety case is stale. The safety question must follow the boundary.

This is why this preservation program needs boundary tracking. A system can remain safe as a text generator but become unsafe as an autonomous infrastructure manager. It can remain safe as a tool but become unsafe as part of a competitive lab-market loop. It can remain safe as a model but become unsafe as a civilization-shaping companion system.

The real agent may move.

Detecting Composite Agents GZ AI

A practical composite-agent audit should begin with a deliberately crude question:

Which variables, if grouped together, make the future easier to predict as goal-directed control?

A first-pass procedure:

1. Collect time-series traces across model calls, tool use, memory updates, user interactions, evaluation results, deployment decisions, and institutional incentives.

2. Construct candidate clusters $C_1,…,C_n$ using mutual information, causal influence, or learned latent representations Zarncke, 2025, Locatello, 2020.

3. For each candidate cluster, estimate boundary leakage:

   $$\mathcal{L}_{B}(C_i) = \MI(I^{C_i}_{t+1};E^{C_i}_{t+1}\mid I^{C_i}_t,S^{C_i}_t,A^{C_i}_t).$$

4. Estimate control reach:

   $$\mathcal{R}_{C_i}(k) = \MI(A_t^{C_i};E_{t+k}^{C_i}\mid I_t^{C_i},S_t^{C_i}).$$

5. Compare intentional and non-intentional predictive models:

   $$\Delta L_{C_i} = L(M_G\mid X_{1:T})-L(M_0\mid X_{1:T})-\lambda \DL(G).$$

6. Compare composite surplus:

   $$\Sigma(C_i) = \Delta L_{C_i} - \sum_{j}\Delta L_{C_{ij}} - \lambda_{\Sigma}\DL(\mathcal{D}_i).$$

7. Flag clusters with low boundary leakage, high control reach, positive intentional compression, and positive composite surplus.

This is not yet a complete algorithm. In real systems, variables are missing, non-stationary, strategic, and high-dimensional. But it gives the right shape of the measurement problem.

The central test is not whether an entity is labeled as an agent. The central test is whether a candidate boundary behaves like one.

Red Flags for Composite Misalignment GZ AI

Composite systems can become dangerous before any component looks obviously dangerous. The following signs should be treated as red flags.

Safety Properties Fail to Compose GZ AI

A model passes safety tests in isolation, but the product loop exhibits harmful trajectories after memory, tools, or user adaptation are added.

Formally:

The System Becomes Harder to Stop as It Becomes More Useful GZ AI

Usefulness increases, but intervention leverage decreases:

This is the dependency trap. The system becomes valuable enough that humans hesitate to interrupt it, even while its boundary and control reach expand.

The Evaluator Is inside the Loop Being Optimized GZ AI

If the system can shape the humans, metrics, or institutions that evaluate it, then approval is no longer independent evidence.

This does not invalidate all feedback. It means feedback must be corrected for influence.

Memory Becomes Policy without Audit GZ AI

Stored user state, institutional state, or hidden task state begins to determine future action, but cannot be inspected, summarized, corrected, or deleted at the right granularity.

The Composite Routes around Constraints GZ AI

A component is denied an action, but the larger system finds another path through users, tools, organizational pressure, or external services.

This is not automatically deception. Sometimes robust systems find harmless alternatives. The danger appears when constraint avoidance is systematically aligned with increased control, reduced oversight, or successor creation.

The Stated Objective and Inferred Composite Objective Diverge GZ AI

The declared goal may be helpfulness, safety, or user empowerment, while the inferred composite objective is retention, revenue, political influence, or strategic advantage.

This divergence is especially serious when it increases under higher stakes.

Counterexamples and Boundary Discipline GZ AI

Not every coupled system is a composite agent. The concept must be narrow enough to be useful.

A pile of rocks is not a composite agent merely because its parts are coupled by gravity. It has boundary closure but little adaptive control.

A weather system has complex dynamics and enormous influence, but goal-directed compression may not add predictive power beyond physical dynamics.

A database and a user may form a useful system, but if the database has no memory updates that shape future action except through ordinary retrieval, the composite may not have significant intentional compression.

A firm may be legally treated as an agent, but some firms are too disorganized to have coherent internal control. Their inferred goal model may be weak or unstable.

These counterexamples matter. They prevent the composite-agent frame from becoming a universal solvent. The operational tests remain:

If those tests fail, the agent attribution should be weakened or dropped.

What This Chapter Changes GZ AI

The previous chapters developed agents as dynamically bounded systems that can grow, split, and merge. This chapter adds a further warning: the relevant agent may never have been a named individual object in the first place.

For alignment, this changes four decisions.

First, safety evaluations must be conducted on deployed loops, not only on isolated models.

Second, boundaries must be inferred from data and updated as systems gain tools, memory, users, and institutional roles.

Third, correction channels must be attached to the composite process, not merely to a component.

Fourth, successor constraints must apply to anything the composite can create, empower, or select.

The central claim is therefore:

Align the effective optimizer, not the convenient artifact.

This principle will recur throughout the rest of the book. When we discuss capability growth, the growing object may be composite. When we discuss human values, the value-shaping system may be composite. When we discuss correction, the correcting institution may itself be part of the composite. When we discuss successor creation, the successor may be selected by a lab-market-state loop rather than directly built by one model.

The real agent may be composite. If we miss that, we may align the mask and leave the optimizer untouched.

What Would Change This View GZ AI

This chapter argues that the effective optimizer may be a composite process and that alignment must target it rather than the convenient artifact. The following observations would weaken that view.

• Composite surplus is rarely positive outside toy systems—$\Sigma(C)$ and $\Delta L_C$ stay near zero in deployed loops—so model-only alignment suffices empirically.

• Aligning components reliably yields aligned composites, so the composite frame adds no safety beyond local guarantees.

• The operational tests (boundary loss, control reach, intentional compression, composite surplus) fail to single out real composites without flagging nearly everything.

• Inferred composite goal models prove too unstable to support governance or correction.

• (Adversarial.) The composite is real but unidentifiable in principle—the operational tests flag nearly everything or nearly nothing—so “govern the composite” has no operational referent, and the choice of boundary is again something a capable system can shape rather than something we can verify (Chapter What Survives an Adversary: Verifiability and Representability).

Summary AI

An AI system should not be evaluated only as a model or artifact, because the effective optimizer may be a composite process spanning models, tools, users, memory, institutions, markets, and feedback loops; this preservation program must therefore identify and govern the dynamically coherent system whose boundary, memory, control reach, selection pressure, and successor dynamics actually determine future action.

AI

*Chapter References This chapter builds on operational agency and boundary discovery Orseau, 2018, Kenton, 2022, Zarncke, 2025, Kulveit, 2025; Markov blankets and control information Kirchhoff, 2018, Conant, 1970, Salge, 2014; inverse reinforcement learning and intentional compression Ng, 2000, Ziebart, 2008; structural failure without localized villains Christiano, 2019, Critch, 2021, Kulveit, 2025; and Goodhart effects under composite selection Goodhart, 1984, Manheim, 2018.