Interview with Nick Bostrom on Superintelligence & Deep Utopia
The interview has now been released! We covered a lot of stuff which at the time of writing hasn’t been covered elsewhere in detail.
Super excited that I’ll be interviewing Nick Bostrom again in a few days – last time was in 2012 (on XRisks of AI and the Simulation Argument). This time we will cover topics that range from Superintelligence to Deep Utopia.
Since Nick Bostrom wrote Superintelligence, AI has surged from theoretical speculation to powerful, world-shaping reality. Progress is undeniable, yet there is an ongoing debate in the AI safety community – caught between mathematical rigor and swiss-cheese security. p(doom) debates rage on, but equally concerning is the risk of locking in suboptimal futures for a very long time.
Zooming in: motivation selection—especially indirect normativity—raises the question: is there a structured landscape of possible value configurations, or just a chaotic search for alignment?
Enter Deep Utopia: not just avoiding catastrophe but ensuring resilience, meaning, and flourishing in a ‘solved’ world; a post instrumental, plastic utopia – where humans are ‘deeply redundant’.
This is our moment to shape the future. What values will we encode? What futures will we entrench?
See below:
Chapters / Timestamps:
0:00 Highlights
1:32 Intro
2:40 Interview – shifts in discourse
6:04 p(doom)? Likelihood that AI will doom us? What counts as doom?
7:34 From dismissal to acknowledgement – geopolitical dynamics surrounding AI advancements and safety measures
9:10 How soon superintelligence?
12:34 AI risk challenges: alignment, political challenges, ethical considerations, and fitting into broader cosmic context (i.e. a society of mature civs/cosmic hosts)
15:06 Cosmic hosts, trust, game theory & utility maximisation: the need for humility in deciding which norms to load into superintelligence
17:55 The dynamics of the landscape of value & initial trajectories effecting where superintelligence ends up
22:19 Maximisation tempered with humility, and distinguishing good from bad values – dichotomy between moral realism and anti-realism
23:55 Satiability vs insatiability
26:43 Metacognition and it’s location in the landscape of value
29:12 AI alignment – control vs motivation
33:34 The Indirect normativity approach to AI alignment (alternative to capability control)
34:01 Elios AI
35:00 Indirect normativity to debug metaphysical, epistemic, metaethical assumptions
40:37 Indirect normativity in practise given the kind of AI we see today
42:58 AI safety: Swiss cheese vs mathematical precision
45:57 Do we need to provide AI with metaphysical/epistemic/metaethical axioms?
50:54 Epistemic deference to AI on concepts too complex for the human mind to grasp
53:29 Does AI understand stuff?
54:56 Human-like AI now b/c training data – perhaps alien-like AI soon based on new training regimes
57:30 Base Camp at Mt Ethics, cosmic norms as metaethics developed by a cosmic host
59:39 AI Creation & the Cosmic Host – train AI to defer to cosmic norms
1:03:20 Values as causal influencers of goals, so better get the values right
1:06:36 Deep Utopia if we get superintelligence right – what is Utopia
1:13:16 Shallow redundancy vs deep redundancy – subtle values coming into focus
1:14:12 Deep utopia may involve giving up some things. Workng through repulsiveness of utopain visions
1:14:39 Going back to basic constituents of values, from there rebuilding value
1:15:36 Philosophical particle accelerator
1:19:12 Taking the scenic route to posthuman forms instead of rushing
1:28:01 Hedonics and the 5 Ringed Defence of Meaning in a plastic utopia
1:40:20 The space of subtle values is enormous, Ikagai and Japanese culture
1:49:45 Maximum novelty experienced early in life?
1:52:04 Putting the picture together
1:55:34 Outriders, and Finding Purpose Now
1:57:04 Reflections on the old era of futurology – Conclusion
