… and the Long First Turn (of the new game). We’re not just designing AI to behave nicely in our human sandbox—but in case the sandbox turns out to be part of a much larger galactic metropolis, we’d better make sure the AI doesn’t start flinging galactic sand in anyone’s eye. We may at some…
No one in their right mind can deny the struggle to survive animates natural selection. Though it’s a bit misleading to say that ‘survival is paramount – and that every strategy serves that goal’: survival plays a crucial role in evolution, but it is not the ultimate goal. As emphasised by Charles Darwin’s theory of…
Value Space and “Good” Human Values Human values can be conceptualised as occupying regions in a vast objective multidimensional “value space.” These regions reflect preferences for cooperation, survival, flourishing, and minimising harm, among other positive traits. If TAI / SI can approximate the subset of human values deemed “good” (e.g., compassion, fairness, cooperation), while avoiding…
Note: this is by no means exhaustive. it’s focus is on western philosophy. I’ll expand and categorise more later… The earliest precursors to indirect normativity can be traced back to early philosophical discussions on how to ground moral decision-making in processes or frameworks rather than specific, static directives. While Nick Bostrom’s work on indirect normativity…
Implications for AI Safety and Indirect Normativity The challenge of understanding value space has moved from a philosophical curiosity to an urgent technical problem. As we approach the development of advanced artificial intelligence, the structure of value and our ability to specify it has become perhaps the most critical challenge facing humanity. This challenge assumes…
Kristian Rönn discusses his amazing book The Darwinian Trap – Macroeconomics & Game theoretic Imagineering: avoiding the minefields of the tragedy of the common in the hope that we can stave off the onslaught of Darwinian Demons, and ultimately find some nash equilibrium of fairness for all humanity, sentient AI an non-human animals. The book…
Will AI Preserve Our Values, or Improve on them? Imagine a future where advanced AIs follow every instruction flawlessly – but humanity feels strangely adrift. Our tools are obedient, but the ‘soul’ of our civilisation feels absent. This is the hidden danger of V-risk—value erosion. In this post I explore what I explore what I…
Some people want to infuse AI with the codification of our existing moral judgements. CEV takes the idea a bit further. In short, CEV is a conceptual approach to aligning superintelligent AI by designing it to act according to an idealised version of what humans would want if they had more knowledge, were faster thinkers,…
The concept of a non-arbitrary objective value space offers a compelling framework for understanding how to navigate to more valuable worlds. This framework presupposes the existence of stance-independent moral truths – ethical principles that rational agents will converge upon given sufficient cognitive sophistication [1]. Human values are a narrow slice of value space – In…
Is self-interest by necessity the default motivation for all agents?Will Superintelligence necessarily become a narcissistic utility monster? No, self-interest is not necessarily the default motivation of all agents. While self-interest is a common and often foundational drive in many agents (biological or artificial), other motivations can and do arise, either naturally or through design, depending…
