AI safety and alignment

The field of AI safety and alignment research asks how to build advanced artificial intelligence such that it remains beneficial to humans as capabilities grow. It ask: how can we keep humans able to correct mistakes and reduce catastrophic or existential risk from runaway AI processes.

Wikipedia’s article on AI alignment:

In the field of artificial intelligence (AI), alignment aims to steer AI systems toward a person’s or group’s intended goals, preferences, or ethical principles. An AI system is considered aligned if it advances the intended objectives. A misaligned AI system pursues unintended objectives.

Alignment is a subfield of AI safety, alongside robustness, monitoring, and control of AI.

Norbert Wiener said in 1960:

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively […] we had better be quite sure that the purpose put into the machine is the purpose which we really desire.

Eliezer Yudkowsky’s version of the complexity of value is:

Any simple goal you try to describe that is All We Need To Program Into AIs is almost certainly wrong.

The field is not one research program but a multitude of agendas: labs, nonprofits, academic groups, governance institutes, and research collaborations that share vocabulary while disagreeing on terminology, near-term priorities, and what would count as success.

A map of the field

Useful starting points:

• AISafety.com map — ecosystem map of organizations, programs, and resources (the clustering on this site rolls up map listings into coherent agenda rows).
• AI Safety Interventions — index of roughly ninety named interventions across foundational theory, oversight, control, interpretability, governance, and underexplored routes; extended PDF.
• AI Alignment: A Comprehensive Survey (Ji et al., 2023) — academic survey of alignment problems and methods.
• Foundational challenges in assuring alignment and safety of LLMs (Ganguli et al., 2024) — Anthropic’s framing of eighteen foundational challenges.
• Open Problems in Technical AI Governance (Reuel et al., 2024) — policy and institutions adjacent to technical work.
• Center for AI Safety — field-building, statements, and course material.
• AI Alignment Forum / LessWrong — long-form research discourse and tag-based archives.
• Human Compatible (Russell, 2019) — assistance-games framing of beneficial AI under preference uncertainty.
• International AI Safety Report — periodic synthesis for policymakers (complementary to researcher-native maps).
• Textbook from the Future (Iliad) — theory megaproject and coordination instrument sketching a communal table of contents for foundational alignment; agenda card.

This site adds an overview of who the major agendas are, the cruxes of the field, and what evidence each has published on the cruxes. The cruxes are presented them as bridges, named conditional handoffs between the cruxes and toward overall alignment. For example, Embedded Agency/MB1 asks whether an agent–environment boundary is sound enough to trust, i.e., the embedded-agency point of the missing clear cut between “the model” and “the optimizer”. None of these cruxes are solved, but different agendas have made progress to different degree on eaach.

What you will find here

1. Agenda cards — one page per research or advocacy program. Browse all via the Field agenda badge.
2. Coverage matrix — The main table of agenda ad bridges below. Column headers link to bridge cards; row headers link to agenda cards. Cell tags link to the evidence catalog below.
3. Evidence catalog — citations backing the matrix.

For term disambiguation across agendas, see the inter-agenda glossary (manuscript App E is synced separately). For how this project maps bridges to field cruxes, see Appendix B.