Intelligence, Motivation and the Future of Value
The central insight here is that intelligence determines what an agent is capable of achieving, while motivation determines what it seeks to achieve. As intelligence increases and external control becomes less reliable, the importance of the agent’s own motivations increases.
- P1: If an autonomous superintelligence exceeds our capacity for reliable external control, its internal motivations will become decisive in determining its chosen ends.
- P2: If such a superintelligence is developed, there is a substantial possibility that it will exceed that capacity.
- C: Therefore, there is a substantial possibility that its own motivations will become decisive.
* In expected-value terms, I think this is a reasonable probabilistic argument, but “substantial possibility” needs formalisation to make the inference rigorous.
For example, let \(A\) mean that external control becomes unreliable (which is something many, if not most, serious AI alignment researchers agree with).
And let \(M\) mean that motivations become decisive.
If we judge that \(P(A)=0.7\) and \(P(M\mid A)=0.9\), then:
\[ P(M)\geq P(M\cap A)=0.63 \]
If the future cannot be reliably controlled by humans, and we remain uncertain about what is ultimately valuable, then we have reason to develop superintelligence motivated not merely to pursue our present conception of the good, but to discover and respond to what is genuinely good.
Astronomical Stakes: What AI Values Will Shape it’s Light-cone
P1. Intelligence amplifies power.
Greater intelligence increases an agent’s ability to shape the world.
P2. Greater capability challenges control.
As autonomous intelligence surpasses ours, reliable human control may become increasingly difficult.
P3. Motivation determines direction.
Intelligence determines what an agent can achieve; its values determine what it seeks to achieve.
∴ Conclusion: As superintelligence exceeds our capacity for control, what it values becomes increasingly decisive in determining the future.
Seeing beyond our current historical vantage point and outside of our human commitment to particular values, perhaps the most important thing to instill in superintelligence a motivated commitment to discovering what is genuinely valuable and acting accordingly.
Super value discovery
Will superintelligence be better positioned to discover value than humans are? Yes. This is plausible, but I would make it conditional on more than intelligence alone.
Imagine three epistemic advantages and one potential disadvantage.
Greater reasoning capacity : A superintelligence might identify subtle contradictions in ethical theories, evaluate complex consequences and explore normative possibilities that exceed human comprehension.
Greater empirical understanding : It might develop much more precise theories of consciousness, valence, welfare, agency and other properties potentially relevant to value.
Exploration of unfamiliar possibilities : It might investigate valuable states of existence that humans cannot readily imagine, including unfamiliar forms of experience, relationships, creativity, cognition and agency.
Potential lack of first-person acquaintance : A non-conscious superintelligence might lack phenomenal experience of suffering, pleasure or other valenced states. Whether it could fully understand their normative significance through third-person investigation remains unresolved.
Intelligence Amplifies Power and Motivation Directs It
Intelligence amplifies an agent’s capacity to shape the world, but it does not, by itself, determine which ends that agent will pursue. As artificial intelligence grows beyond human capabilities, our ability to control its actions through external constraints may become increasingly unreliable. As such, the agent’s own motivations and values become increasingly consequential.
This makes motivation a central problem of superintelligence alignment. But which motivations should we seek to instil?
Humanity has some knowledge of what matters, particularly through our acquaintance with conscious experience, suffering, flourishing and agency. Yet our understanding of value remains incomplete, contested and potentially limited by our cognitive architecture.
If there are objective normative truths, then a sufficiently capable superintelligence may be better positioned than humanity to discover them. But greater intelligence does not guarantee wanting to discover normative/moral insight, nor does such an insight guarantee appropriate motivation.
Many people worry about loss of control of AI and that AI won’t align to their present values. We should be concerned with the grand challenge of developing AI with an enduring motivation to discover and respond to what genuinely matters, while retaining epistemic humility, practical wisdom and corrigibility to good reasons.
The ultimate alignment challenge shouldn’t be about ensuring that superintelligence preserves our present values – it should be more about ensuring that it can discover better ones without destroying what already matters along the way.
Call to Action
Here we expand the syllogism and break it down into three connected stages. It combines factual premises with a normative premise about what ought to be done.
Stage A: The Power and Motivation Argument
P1. Intelligence, combined with effective agency and access to resources, increases the capacity to shape the world.
P2. As an autonomous superintelligence’s capabilities grow, relying exclusively on external human control may become increasingly unreliable.
P3. An autonomous agent’s motivations principally determine the ends it pursues.
C1. Therefore, under conditions where reliable external control is limited, a superintelligence’s motivations become increasingly consequential in determining how it uses its power.
Stage B: The Normative Discovery Argument
P4. Humanity’s understanding of what is valuable is incomplete and potentially mistaken.
P5. If discoverable normative truths exist, agents with greater reasoning and investigative capabilities may be better positioned to discover them, provided they are appropriately motivated and epistemically competent.
P6. Prematurely fixing a superintelligence’s ultimate values to our current understanding risks entrenching our normative mistakes.
C2. Therefore, we have substantial reason to investigate motivations that support continued normative learning and revision, rather than exclusively attempting to preserve a predetermined conception of the good.
Stage C: The Practical Alignment Argument
P7. We ought to take proportionate, feasible steps to reduce the risk of catastrophic harm and improve the prospects for morally valuable futures.
P8. Developing reliable motivations, normative learning, corrigibility and practical competence are plausible ways to reduce those risks, though their effectiveness requires empirical validation.
C3 – Call to action: We should prioritise research into AI architectures that are motivated to discover and respond to what genuinely matters, remain corrigible to good reasons, and preserve opportunities for normative learning, while maintaining effective capability controls and governance.
