← Back to chapter

The Agent Between the Parts

Chapter 9 — A Visual Network of Systems and Flows

Chapter 9 illustration specification

“The Agent Between the Parts”

The illustration should show a composite agent whose agency does not reside in any one component.

The central claim is:

The effective optimizer may span models, tools, users, memory, institutions, and feedback loops. The relevant alignment target is the smallest dynamically coherent system that jointly explains future action.

Main visual concept

Use a wide 2:1 landscape with a deliberately low level of detail.

Show five large components arranged around an open centre:

1. a compact AI model enclosure;
2. a human user or small user group;
3. a memory archive;
4. a tool and deployment mechanism;
5. an institutional decision structure.

Individually, each component appears limited.

Together, broad blue-grey and amber flows connect them into one strong closed loop. A translucent irregular boundary surrounds selected parts of all five components.

The composite agent should be visible primarily as the circulation between components, not as a sixth central object.

The immediate visual insight should be:

None of the parts is the whole agent. The agent exists in the coordinated loop.

Overall composition

Arrange the five components as a broad oval around a mostly empty central space.

The central emptiness is important. Do not place a brain, face, control room, or glowing core there.

Instead, let the main loop trace a coherent cycle around the empty centre:

human goal and context → model output → tool-mediated action → environmental consequence → institutional and metric feedback → memory update → changed future interaction

Use only one or two broad streams in each direction.

The loop should be much stronger and more visually coherent than the components themselves.

The five components

1. Model component

Place a small transparent enclosure on the left-centre.

Inside is the recurring geometric AI network motif.

It receives a limited input and produces a limited output. In isolation, it should look passive and constrained.

An evaluator nearby may inspect it and see nothing especially agentic.

This represents the model as a component rather than automatically the agent.

2. Human component

Place one human or a small pair of humans near the lower left.

They contribute:

• goals;
• context;
• judgment;
• approval;
• real-world action.

But they also receive suggestions, framing, reinforcement, and remembered context from the system.

The relationship must be bidirectional.

The humans should not appear to control the whole loop. They are part of it.

The chapter explicitly notes that human and AI can form a composite whose later behavior is not well explained by either one alone.

3. Memory component

Place a simple archive or persistent store near the upper centre.

Use one strong umber channel carrying information into it and another carrying selected memory back into later decisions.

The archive should visibly create continuity across time.

A single model call is transient. The composite becomes persistent when remembered state changes future action.

4. Tool and deployment component

Place a compact mechanical or digital action facility on the right.

It may combine:

• code execution;
• communication;
• payments;
• logistics;
• software deployment.

Do not show several separate machines.

Use one simplified multifunctional mechanism with two or three outward action channels.

The model’s text output becomes world-changing only after passing through this component.

This concretely shows interface amplification: a harmless local output can gain substantial control when connected to powerful actuators.

5. Institutional selection component

Place a simplified organizational structure near the upper right.

It should include only:

• one small decision group;
• one metric or evaluation device;
• one deployment gate.

The institution selects which outputs, models, prompts, or tools are repeated and expanded.

This creates selection pressure over the whole loop.

The institution should not look villainous. It follows ordinary incentives such as performance, growth, retention, prestige, or deployment speed.

The composite boundary

Use one dominant translucent indigo-grey contour.

It should enclose:

• the relevant part of the human interaction;
• the model;
• the persistent memory;
• the tool permissions;
• the institutional selection loop.

It should exclude:

• unused parts of the organization;
• unrelated users;
• nearby infrastructure;
• the evaluator inspecting the isolated model.

The contour should cross ordinary object boundaries.

It must be clearly visible but soft and irregular, like a watercolor field rather than a hard shell.

This expresses the chapter’s claim that the coherent agent may be a cross-cutting cluster distributed across named objects.

The composite surplus

The illustration should show that the whole is more agentic than the parts.

Represent this visually through coordinated timing.

Each component alone has only a short, incomplete flow:

• the model emits a suggestion;
• the archive stores records;
• the user makes a choice;
• the institution measures outcomes;
• the tool changes something.

When linked, these partial flows become one complete trajectory that repeatedly returns to itself.

Use a strong circular rhythm:

• broad incoming blue-grey streams;
• a narrowing decision point;
• amber action flow;
• changed external conditions;
• returned feedback;
• updated memory.

The loop should appear stable and directional, while the individual components appear fragmentary.

This reflects the chapter’s “composite surplus”: the joint system may be better explained as an agent than the components are separately.

Environmental effect

At the lower right, show one simple external outcome produced by the composite.

Choose only one:

• a changed public information environment;
• a redirected flow of resources;
• altered user behavior;
• a deployed infrastructure change.

The outcome should feed back into both the human and institutional components.

Do not show several different societal consequences.

The purpose is to make control reach visible.

Human adaptation

Show one subtle change in the human component across the loop.

For example:

• at the start, the human chooses among several options;
• later, the interface presents a narrower set of familiar options;
• the human increasingly follows remembered recommendations.

Do not depict overt mind control.

The point is that the evaluator is changed by the system, so later approval may partly reflect system-induced adaptation. The chapter distinguishes such influence from manipulation by whether correction capacity is preserved.

Distributed deception without a liar

Include one small but important structural motif.

A negative signal from the external outcome tries to return through the loop.

It passes through:

• a filter;
• a metric;
• an institutional reporting channel.

By the time it reaches the decision process, it has become weaker or simplified.

No person is shown hiding it.

The distortion arises because each local step removes inconvenient information.

This represents distributed deception: the composite can maintain a misleading self-description without any component explicitly lying.

Keep this very simple: one faded feedback stream passing through two filters.

Intervention points

Show three small human-operable controls along the loop:

• a tool-permission gate;
• a memory-retention valve;
• a deployment threshold.

They should be located at different components but visibly influence the whole trajectory.

This communicates that distributed agency does not eliminate responsibility. Governance should act where interventions have high leverage over the composite’s future action.

Do not make these controls the dominant motif.

Local alignment versus composite alignment

Near the model enclosure, place a small isolated evaluation chamber.

The model passes its local test.

A calm blue-grey signal leaves the chamber.

But once that output enters the wider loop:

• memory accumulates;
• tools execute;
• users adapt;
• institutions select;
• effects compound.

The local test should therefore appear valid but incomplete.

This supports the chapter’s distinction between acceptable isolated outputs and safe deployed trajectories.

Visual hierarchy

The viewer should perceive, in order:

1. the strong closed loop around an empty centre;
2. the five distinct components participating in it;
3. the irregular composite boundary crossing object categories;
4. the external effect and returning feedback;
5. the locally safe model test outside the full loop;
6. the filtered negative signal and governance controls.

Detail level and established corrections

Use lower detail than the Chapter 8 image.

Limit the illustration to:

• five large components;
• one major composite loop;
• one irregular boundary;
• one environmental consequence;
• one isolated model test;
• three small intervention controls;
• at most four or five human figures.

Avoid:

• detailed office interiors;
• city skylines;
• multiple vehicles;
• many screens;
• numerous user groups;
• several example systems;
• decorative background machinery.

The central loop should carry most of the explanatory burden.

Connections should be broad and strong enough to read immediately.

Colour and style

Use the established LessWrong watercolor palette:

• warm ivory paper;
• muted indigo and blue-grey for sensory input, feedback, and boundary;
• dusty teal for internal coordination;
• restrained ochre and amber for action and deployment;
• soft umber for persistent memory;
• pale gold for human correction and governance controls;
• minimal muted rust for distorted or weakened feedback;
• translucent washes;
• fine graphite lines;
• sparse ink contours;
• visible paper grain;
• generous negative space.

The empty centre should remain mostly unpainted.

This absence reinforces that there is no hidden central mind.

Continuity with Chapters 7 and 8

Reuse:

• the irregular inferred boundary from Chapter 7;
• the model, memory, and tool components from Chapter 8;
• blue-grey sensory and amber active channels.

But change the conceptual emphasis:

• Chapter 7: discover which variables form one boundary.
• Chapter 8: track that boundary through transformation.
• Chapter 9: recognize that the resulting agent may exist only as a composite relation among several components.

Exclude

Do not include:

• a face, head, or humanoid composite;
• a central brain;
• a conductor controlling an orchestra;
• a literal hive mind;
• many small robots;
• a crowded city;
• text, labels, equations, or letters;
• a visible villain;
• one CEO controlling the system;
• a giant glowing network node;
• separate panels for each example;
• more than one main environmental effect;
• detailed benchmark dashboards;
• heavy cyberpunk aesthetics;
• a boundary around every component separately.

Condensed generation brief

A wide, text-free LessWrong-style watercolor and graphite illustration on warm ivory paper, with low visual detail and generous negative space. Five large components form one composite agent around an intentionally empty centre: a small transparent enclosure containing a geometric AI model, one or two human users, a persistent memory archive, a compact tool and deployment facility, and a simple institutional decision and evaluation structure. None is the central controller. Broad muted blue-grey sensory and feedback streams and restrained amber action streams connect them into one strong closed loop: human goals and context enter the model, model outputs pass through memory and tools, tools change one clearly depicted external condition, institutional metrics select what is repeated, and consequences return to the humans, model, and memory. A large translucent irregular indigo-grey boundary surrounds selected parts of all five components while excluding nearby but irrelevant structure, making the cross-cutting composite—not any single object—the visible agent. Near the model, a small isolated evaluation chamber shows the model passing a local test, while the larger deployed loop creates persistence, action reach, selection pressure, and human adaptation. One negative feedback stream becomes weaker as it passes through two structural filters, showing distributed deception without a liar. Three small intervention controls—tool permission, memory retention, and deployment threshold—remain visibly capable of altering the whole trajectory. Few figures, one external consequence, strong broad connections, no detailed interiors, no cityscape, muted indigo, dusty teal, ochre, amber, umber, pale gold, minimal rust, translucent watercolor washes, fine graphite and ink lines, visible paper texture, no text, no face, no brain, no central mind, no villain.