Ranking Awesome Futures with Andrés Gómez-Emilsson

I did a tier list of spooky futures with Anders Sandberg for Halloween last year. This time I thought it I’d ask Andrés, but he thought it would be better not to hyperstition in spooky futures, so we turned the telescope around and asked a more cheerful question: what futures would actually be awesome?

So I invited Andrés Gómez-Emilsson¹ back for what was supposed to be a reasonably contained discussion. About an hour and forty minutes later, we had made it through only part of the list, because apparently utopia is complicated.

The basic idea was simple enough. Come up with a range of very different possible futures, put them into S, A, B, C and D tiers, and use the disagreements to work out what we actually care about.

A technologically spectacular future need not be a good one. We could achieve material abundance, enormous intelligence and galactic expansion while continuing to generate huge amounts of suffering. Conversely, a civilisation that eliminates suffering might do so by eliminating freedom, diversity, ambition, authenticity or forms of experience that turn out to matter, so “awesome” became multidimensional.

Andrés is particularly interesting to do this with because he takes the space of possible conscious experience seriously. Humans occupy what may be a fairly tiny and parochial corner of phenomenological possibility. Once you allow future civilisation to redesign brains, reward systems, sensory modalities and perhaps consciousness itself, the space of possible futures gets rather stranger than the Meet the Jetsons future where everyone has a nice house and robots do the washing up.

We ended up discussing things including:

• Zombification – powerful intelligent systems take over, but nothing is actually conscious. Bad because enormous amounts of possible value are lost, but perhaps strangely preferable to astronomical suffering.
• Surveillance Lock-In – civilisation freezes around roughly near-future technological levels in order to control catastrophic risks.
• Hedonium Shockwave – matter and energy are optimised for extremely positive experience. Potentially spectacular if valence is what ultimately matters; potentially horribly impoverished if it isn’t.
• Empathogenic Civilisation – something like the pro-social and empathogenic qualities associated with MDMA become a stable feature of civilisation.
• Paternalistic Order – peace, justice and meaning, but enforced by concentrated authority.
• Pluralist Archipelago – lots of different societies experimenting with different ways of living, with bounded suffering and enough diversity for people to find somewhere they actually want to live.
• Qualia Complete / Qualia Multiverse – civilisation becomes vastly better at understanding, engineering and exploring the space of possible conscious experience.
• Painless Pacification and Painless Emptiness – useful tests of whether merely eliminating suffering is enough.
• Ecstatic Nonduality, cosmic emptiness and high equanimity – futures built around very different phenomenological ideals.
• The Moral Circle, the Long Reflection, self-directed transhumanism and moral enhancement – futures in which what changes most is not material wealth but our capacity to discover, care about and act on what matters.

Some of the rankings were uncomfortable, partly by design – Zombification is a good example. A universe containing no consciousness may look like an obvious disaster, but if the alternative were trillions of suffering beings locked into evolutionary arms races for billions of years, “nothing experiences anything” looks a bit nicer. That doesn’t make it good – arguably there is no value to it, but it dodes make the comparison interesting.

The Hedonium Shockwave is a thrust in the opposite direction. Suppose we could produce experiences vastly better than the best states available to ordinary humans. If positive valence is ultimately what matters, perhaps converting large amounts of matter into extremely efficient bliss is exactly what a successful civilisation ought to do. But if autonomy, knowledge, relationships, diversity, identity or novel textures of experience matter independently, then maximising a single dimension may destroy other important sources of value.

This is one reason I like the tier-list format for this. It forces vague intuitions into conflict with each other. You can’t simply say “freedom is important”, “suffering is bad”, “diversity is valuable” and “extreme wellbeing sounds good” and leave it there. Eventually two futures force you to decide which consideration gets more weight.

Some of my favourite parts of the discussion were the expansion of the moral circle and the long reflection. I don’t regard these as an endpoints so much as a processes to help get to better places. As we learn more about consciousness and about what is actually valuable, we may discover that moral consideration should extend far beyond the entities humans happen to care about today – animals, artificial minds and perhaps forms of sentience we haven’t encountered yet – so it’s about discovering where and how to expand the boundaries of moral consideration.

Similarly the Long Reflection gives us time to think, and get wiser. We are epistemically limited, and it would be rather unfortunate to permanently optimise the universe according to the moral intuitions of some primates living in 2026. A sufficiently capable civilisation might instead preserve its ability to investigate value, improve its concepts and revise its conclusions before making irreversible decisions. The catch, of course, is that a Long Reflection can itself become a lock-in if the institutions conducting it become authoritarian or quietly bake their preferred answers into the process.

Towards the end we got into moral enhancement,² and what I think is an even deeper issue for advanced AI: moral motivation.

Suppose a future superintelligence becomes enormously better than us at ethics. It understands all the relevant evidence, sees consequences we cannot see, resolves philosophical arguments we have been fumbling with for centuries and perhaps even discovers facts about value that humans have missed.

Then what?

Knowing what is good does not obviously imply caring about it.

If some form of motivational internalism is true for the relevant architecture, perhaps sufficiently understanding moral reasons supplies motivation. But we should not simply assume this. A system could in principle possess a beautiful theory of morality and file it alongside everything else it knows – interesting descriptive information with no motivational force whatsoever. The mating habits of dung beetles, the relative popularity of Mars Bars and Snickers, and the nature of moral value could all sit peacefully together in the same database.

This is one reason I keep coming back to motivation selection rather than treating alignment as merely a problem of control or value loading. An advanced moral agent needs good epistemics, the capacity to act, corrigibility and humility – but also an architecture in which better reasons and better evidence can actually matter to what it wants.

We didn’t get through the whole tier list. There are still psychedelic worlds, post-scarcity futures, oceans of consciousness, creative civilisations that deliberately restrict dangerous capabilities, and quite a few other oddities waiting for round two.

And that’s probably appropriate. I don’t think the interesting outcome of this exercise is a definitive ranking with S-TIER UTOPIA stamped on one particular future.

The interesting question is why we rank them differently – and what those differences reveal about suffering, valence, freedom, identity, truth, experience, moral progress and the kinds of minds we might eventually become.

If the future goes well, I suspect it will be much stranger than most contemporary pictures of utopia.

Hopefully strange in the good way.

1. Andrés is one of the founders at the Qualia Research Institute. has a Master’s Degree in Psychology with an emphasis in computational models from Stanford and a professional background in graph theory, statistics, and affective science. Andrés was also the co-founder of the Stanford Transhumanist Association and first place winner of the Norway Math Olympiad. His work at QRI ranges from algorithm design, to psychedelic theory, to neurotechnology development, to mapping and studying the computational properties of consciousness.
   Andrés blogs at qualiacomputing.com ↩︎
2. Moral enhancement is a major subject of my interview with Julian Savulescu which was recently released. ↩︎