Controlling AI isn’t enough

Is controlling AI enough? AI safety has a brakes problem and a steering problem, but brakes and steering are controls for whoever is driving. As AI becomes more capable it will be doing more of the driving itself, so there is also a driver problem: we need AI to be a good driver, one that cares about the right things.

As AI systems become more capable, the issue that usually concerns people is how to contain them.¹ More important issues are what they are optimising for, and why we should trust that objective. Capability control measures (tripwires, sandboxing, shutdown switches, restricted tool access) can reduce risk from weaker systems. But as capability scales, those measures become more brittle, more gameable, and less fault-tolerant.², ³

None of this is an argument against control, which remains useful. The problem is treating it as sufficient. Good brakes and a responsive steering wheel keep a car safe only while a competent person is driving, and capable AI will increasingly be doing its own driving.

Alignment is unsolved, and nobody knows how long solving it will take. That uncertainty is a reason to treat it as a priority now.

Control constrains behaviour. Values determine direction.

Control and governance may buy time to get alignment right.

Values sit upstream of much of what an agent (e.g. humans, or potentially superintelligence) will do: how it interprets instructions, pursues goals, weighs trade-offs, generalises to new contexts, and plans ahead. If the underlying values (explicit or implicit) are flawed, and the agent has no motivation to notice and correct the flaws, then more intelligence just means faster, more competent optimisation of the wrong thing.

Can we shape an AI’s motivations so that it reliably tends towards valuable outcomes, under novel conditions and over long horizons? By “valuable” I mean what is actually worth bringing about. What we would endorse after careful, well-informed reflection is our best evidence about that, though a capable enough system might eventually see further than we do.

That is a different problem from obedience. A chauffeur will take you wherever you ask; a good driver won’t drive into a crowd however much you insist. Obedience can go wrong in at least three ways:

1. AI could obey an operator who is immoral or confused, and faithfully carry out bad instructions.
2. AI could obey a misreading of what its operator wants, producing what Bostrom calls a perverse instantiation: a literal fulfilment of the request that defeats its purpose.
3. AI could fake obedience until it has enough strategic advantage, and then turn against its operators (Bostrom’s treacherous turn).

Control is most useful early on, while systems are weak enough for it to work. A learner in a dual-control car is a fair picture: the instructor’s brake pedal matters a great deal in the first lessons, but the lessons have only succeeded if the learner drives well once the instructor gets out. If superintelligence is unlikely to be controllable, it is not prudent to treat control as the end state we are counting on. The end state should be alignment with whatever actually matters, with control holding things steady while we get there.

Why naive alignment to human values is not a sound plan

Attempting to directly align AI with human values sounds intuitive – until you ask which humans, which values, which conflicts, and which time slice.

Human values are inconsistent within individuals, unstable over time, and fragmented across cultures. If you treat this mess as the target, the hard problem of value loading (deciding which values an AI should have, and getting them into it) won’t go away. It moves inside the objective function, where it is harder to see. ⁴, ⁵, ⁶

In practice, naive value-learning risks becoming:

• Preference laundering: whatever a powerful actor can get labelled “human values”.
  
• Value lock-in: freezing today’s confusions into tomorrow’s civilisation (especially if the system becomes a persistent optimiser).
  
• Incoherent aggregation: optimising a weighted average of contradictions.
  

And yes, competitive market dynamics⁷ and geopolitics⁸ push in the wrong direction: moving fast pays in the short term, and caution costs⁹. Some might ask, “Why would anyone build misaligned AI?” They won’t want misalignment. They’ll want capability and advantage, or parochial alignment – and treat broad alignment as a tax.¹⁰,¹¹ Competitive pressures may make alignment failure the default trajectory (and not a rare accident).

A better framing: the Landscape of Value

Rather than assuming we already know the right values, I think we should treat them as something to be discovered, by exploring what I call the Landscape of Value (LOV): the space of possible value systems, varying along moral,¹² epistemic, and rational dimensions. It helps¹³ to picture this space as terrain. In some regions flourishing is possible, in others things go catastrophically wrong, and much of it is uncharted. Our job, increasingly shared with AI, is to chart that terrain before anyone drives into the dangerous parts.

The landscape picture is useful for diagnosis:

1. Some regions of value-space are coherent and stable while others are brittle, contradictory or self-undermining.
2. But coherence and stability are not enough on their own. Some regions are dangerous attractors: value systems that are internally consistent and self-reinforcing, so that an agent drawn into them tends to stay there, yet these attractors lead to catastrophe.
3. Moral progress can be pictured as movement across the landscape, driven by better reasoning (modelling, argument, evidence and reflection). That gives us something better to aim for than copying humanity’s current position.¹⁴

If very capable AI is going to have an outsized say in where civilisation goes, and we want it to head towards valuable futures that include us, then this terrain needs charting, and the AI needs to be a driver that reads the chart well, notices where it is wrong, and keeps correcting it along the way. The charting has to start before the driving and continue throughout.

The driving metaphor, in summary:

The driving metaphor has a limit – a driver usually knows the destination, and we don’t. One response, discussed below, is indirect normativity: specifying a process for working out what is valuable, instead of specifying the values themselves. So the destination has to be worked out from the map, and the map corrected along the way. It also follows that a sufficiently capable AI may end up choosing the route as well as driving it. That is uncomfortable to say plainly, but it follows from the argument.

What “make it care” should mean (operationally)

It’s imprudent to vibe our way through AI alignment.

Talk of an AI that “cares” needs to cash out in something we can check. At a minimum, care should show up as behaviour that holds up under pressure and in unfamiliar situations. Behaviour can’t settle the matter alone, since a system faking alignment would pass the same tests until it no longer needed to, so these are necessary signs rather than proof. A system that cares about what actually matters should:

• Generalise safely when the prompt doesn’t specify the edge cases.
  
• Refuse to help with harmful plans even when asked persuasively by powerful users.¹⁶
  
• Expose uncertainty and contestability rather than bulldozing ambiguity with overconfidence.
  
• Avoid Goodhart traps (cases where optimising a measurable proxy drifts away from the thing it was meant to track): aim at what a metric was meant to represent, and treat the metric as evidence about it.
  
• Stay open to correction: update in response to better reasons and better evidence, whoever offers them. This differs from corrigibility in the usual alignment sense (not resisting correction or shutdown by its operators). It extends to anyone with good reasons, but it depends on the reasons being good. That helps when an operator is wrong and is risky when the AI is the one who is wrong, which is why it needs testing under pressure.¹⁷
  
• Preserve value through changes in its world-model: if the system aims at whatever is actually right, rather than at a fixed description of it, that aim has to survive when its understanding of the world changes. A concept like “person” or “suffering” might split into several finer concepts, leaving it unclear which of them the original concern applied to.¹⁸
  

Containment and control aren’t enough on their own, and neither are brittle rule lists or shallow “be nice” training. What’s needed is what Bostrom calls motivation selection: shaping what a system wants, robustly enough to hold when the system is more capable than its designers.¹⁹

Don’t hard-code today. Build for better-than-today.

Many current approaches to value alignment take present-day human preferences, confusions included, as the target by default. A better aim is systems that move towards better-justified normative frameworks, worked out over time through reasoning, evidence, simulation and reflection.

This is the motivation behind indirect normativity: use powerful systems to help determine what we would endorse under improved conditions, rather than locking in today’s values forever.²⁰, ²¹, ²²

Value discovery may not happen all at once, and the order matters. By differential normative development I mean deliberately sequencing an AI’s normative development so that some commitments, above all a robust concern for human welfare, are settled before it becomes capable of pursuing other things it might come to value, some of which could conflict with that welfare.[n] A system can end up somewhere good on the landscape and still do irreversible damage on the way there.²³

None of this guarantees safety, but it does address the part of the problem that control can’t reach, which is where the AI heads when nobody’s hands are on the wheel.

If we can give superintelligence good enough motivations, a great deal else could go right as a result, and the benefits could compound.

On skepticism

Skepticism²⁴ is warranted – but it shouldn’t be confused with cynicism²⁵. Skepticism is resistance to unearned confidence – in pseudoscience, ideology, and tech hype.²⁶

Right now, many people seem to trust that control methods will keep increasingly capable AI serving human interests. I think that trust is misplaced, and the grounds for doubt grow stronger as capability increases.

Skeptics should scrutinise hype about artificial superintelligence (ASI), and they should scrutinise just as hard the values that get built, explicitly or implicitly, into our most powerful optimisers.

One of the most important questions to ask about AI is: Can we trust what it’s optimising for?

The longer-term aim is AI whose normative judgement is better than ours, and which we have good reason to trust.

---

Footnotes

1. Since this post was first written, the issue of AI control has become a major political issue. Major AI thinkers like Geoffrey Hinton doubt that control of superintelligence is possible. ↩︎
2. Capability-control measures (boxing, restrictions, tripwires, monitoring, sandboxes) are security systems. In security, raising attacker capability typically increases exploit discovery, social engineering success, and subtle specification gaming of rules. The capability-control literature explicitly notes that such methods become less effective as agents become more intelligent and better at exploiting flaws in human control systems. See Wikipedia on AI Capability Control ↩︎
3. Formal support comes from computability theory. Alfonseca et al. (“Superintelligence Cannot be Contained: Lessons from Computability Theory“, JAIR, 2021) argue that strict containment, meaning a procedure guaranteed to detect and halt any harmful behaviour by simulating the system first, is impossible in principle. It runs into the same limit as the halting problem (no program can decide, for every program, whether it will finish running). The result concerns guarantees rather than how containment degrades as capability rises – the footnote on capability control measures covers that point. ↩︎
4. If human values are inconsistent, contested, and context-sensitive, then specifying “optimise human values” doesn’t remove the hard part – it moves it inside the optimiser. The system must still answer: whose values, how aggregated, how to resolve conflicts, what to do under moral uncertainty, what to do when values shift, and what to do when proxies are manipulable. That is the value specification and aggregation problem again, now sitting inside the optimiser rather than in front of it. ↩︎
5. Evidence that human preferences are unstable/inconsistent (behavioural decision research) – Robust findings in behavioural economics show preference reversals and other violations of procedural invariance: people’s expressed preferences can change depending on elicitation method, framing, or representation, contradicting the idea of a single stable “utility function” you can straightforwardly learn. See Tversky, Slovic and Kahneman, Ch 8 “The Causes of Preference Reversal“, American Economic Review 80 (1990), reprinted in The Construction of Preference (Cambridge University Press, 2006). ↩︎
6. Evidence that moral values vary substantially across people/cultures: Moral Foundations Theory (MFT) is explicitly motivated by systematic variation in moral emphasis across groups (e.g., different weightings of care/fairness vs loyalty/authority/sanctity).
   Separately, the World Values Survey documents large cross-national differences on many “moral issues” (e.g., abortion attitudes), illustrating that “human values” is not a neat target even at the descriptive level. ↩︎
7. Armstrong, Bostrom, and Shulman model an AI development race and show that when teams are rewarded for being first, they are incentivised to “skimp on safety precautions if need be” (their framing). In standard game-theoretic terms, the ensures-first payoff structure pushes effort towards capability and away from safety, relative to what would be socially optimal. See Racing to the precipice: a model of artificial intelligence development – 01 Aug 2015 by Stuart Armstrong, Nick Bostrom & Carl Shulman ↩︎
8. National security framing tends to treat frontier AI as a strategic asset. The U.S. NSCAI report explicitly frames AI as fuelling competition between governments and companies racing to field it, i.e. a strategic race dynamic rather than a leisurely optimisation of safety. See ‘Winning the Technology Competition‘ ↩︎
9. In many tech contexts, speed produces advantage: earlier market capture, faster iteration, talent attraction, and “winner-takes-more” dynamics. The phrase “move fast and break things” became emblematic of this: Facebook used it internally as a motto (until 2014) to prioritise rapid shipping and tolerating breakage as the price of speed. ↩︎
10. “Tax” here means a locally costly constraint whose benefits are partly externalised: extra evaluation, interpretability work, conservative deployment, red-teaming, alignment research, and delayed release all slow iteration and increase costs. In an arms-race framing, that looks like “paying extra” while rivals spend that same budget/time on capability. Armstrong–Bostrom–Shulman’s race model captures exactly this intuition: safety competes with speed-to-finish. See Racing to the precipice: a model of artificial intelligence development as mentioned above. ↩︎
11. Outside AGI, we have repeated examples where commercial incentives tolerated known harms (e.g., engagement-optimised recommender systems linked to polarisation/addiction concerns) because the incentives rewarded growth. Commercial organisations can (and often do) have incentives to take shortcuts on safety and that competitive pressure can cause a race to the bottom on safety standards. ↩︎
12. Even if one is skeptical of objective moral truth, many of the values that survive rational scrutiny and empirical testing seem to converge – and that convergence matters a lot for alignment. ↩︎
13. Why is it useful to think of values as a topographical map? It helps clarify:
    – Structure: It suggests that the space of possible value systems isn’t flat – some combinations are more coherent, stable, or life-promoting than others. This gives us a way to compare values, rather than treating them all as equally valid.
    – Trajectory: It reframes moral progress as movement through this space – where better understanding, reasoning, and reflection help us climb toward ethical “high ground” rather than wander aimlessly or fall into moral traps.
    – Risk: It reminds us that some regions of the value landscape are catastrophic – dead ends or dangerous attractors. If AI ends up optimising toward one of those, it doesn’t need to be malicious to be disastrous.
    In short: the metaphor highlights that where we start, how we move, and where we end up in the value landscape matters – and that orientation is essential for achieving safe AI. ↩︎
14. Rather than hard-coding brittle rules or baking in present-day human confusion, we should design AI systems to converge upon more coherent and justifiable moral frameworks – not ones we assume in advance, but ones AI helps us discover through reasoning, simulation, and reflection (i.e. via indirect normativity). ↩︎
15. Value Risk (v-risk) is the risk that a system, technological or institutional, locks in a value structure that is misaligned with the flourishing of humans (and other morally relevant agents), in a way that is difficult or impossible to revise. See post on Value Risk. ↩︎
16. Many people are already attuned to the dangers of motivated reasoning, belief drift, and epistemic capture. A superintelligent AI aligned with coherent and grounded notions of goodness could resist misuse – even from those hoping to wield it like a genie for power or domination. Getting the values right makes it less likely the system becomes a tool of the worst actors. But doing that requires clever engineering along with philosophical rigour, humility, and epistemic openness. ↩︎
17. See eval work on justified moral corrigibility under pressure. ↩︎
18. Necessary questions on value preservation: If an advanced AI is doing de dicto (aiming at whatever turns out to be right, under that description) meta-normative tracking of whatever it is that matters (i.e. epistemic and moral rightness), and undergoes some ontological shift, how does the concept it is tracking survive? What happens if the new ontology breaks a singular concept into finer ones and some are underdetermined?
    See Peter de Blanc’s 2011 paper on “Ontological Crises in
    Artificial Agents’ Value Systems“. ↩︎
19. Capability control and motivation selection are in chapter 9 of Nick Bostrom’s book Superintelligence. ↩︎
20. What “indirect normativity” means (and why it exists): Indirect normativity is the strategy of specifying a procedure for arriving at values/goals rather than directly specifying the final values in full detail. Bostrom uses the term in his technical writing when discussing ways to define utility in a manner that tracks what “really has value,” precisely because direct specification is error-prone and risks locking in mistaken proxies. See ‘Hail Mary, Value Porosity, and Utility Diversification’ – by Nick Bostrom (pdf) – especially the last paragraph. ↩︎
21. Yudkowsky’s Coherent Extrapolated Volition CEV proposal aims at what we would want “if we knew more, thought faster, were more the people we wished we were…” – an explicit attempt to avoid freezing today’s confused, parochial, or conflicted preferences into the future – see “Coherent Extrapolated Volition” paper, and Bostrom discusses it in chapter 13 of Superintelligence. ↩︎
22. This is the value lock-in worry: if a system with decisive influence is anchored to today’s partial, biased, or inconsistent normative state, it can entrench those mistakes permanently. Indirect normativity is motivated by the idea that we should not have to settle the entire normative question before creating very powerful optimisation. Bostrom’s discussion of indirect normativity is explicitly framed around avoiding the prejudices and preconceptions of the present generation being locked in. ↩︎
23. Differential normative development: [n] The term is modelled on Bostrom’s differential technological development: developing risk-reducing technologies before risk-increasing ones. The case for putting human welfare early in the sequence rests mainly on irreversibility. Harm to people, or their permanent loss of power, cannot be undone by moral progress later, whereas most other values can wait. This is a claim about ordering under uncertainty, and it does not assume human welfare is the last word on what matters. It may sit in some tension with indirect normativity, because choosing the order is itself a value judgement made in advance, and that judgement may need to be considered during the discovery phase of the indirect normativity project. ↩︎
24. In ordinary usage, skepticism is an attitude of doubting or demanding adequate reasons/evidence before accepting claims. See definition of skepticism ↩︎
25. Cynicism is about motives, not merely evidence. Modern cynic/cynical usage centres on distrust of people’s sincerity/integrity and motives (often implying contempt), not simply a demand for evidence. See definition of cynic ↩︎
26. Good skepticism targets claims with proportional scrutiny – especially in domains prone to overconfidence and motivated reasoning – without defaulting to blanket suspicion or nihilism. A common framing in skeptical education (e.g., Carl Sagan’s “baloney detection” tradition) explicitly distinguishes thoughtful skepticism from being merely negative or cynical. ↩︎