… and the Long First Turn (of the new game). We shouldn’t just design AI to behave nicely in our human sandbox in case the sandbox turns out to be part of a much larger galactic metropolis. In this case we’d better make sure the AI doesn’t start flinging galactic sand in anyone’s eyes. We…
No one in their right mind can deny the struggle to survive animates natural selection. Though it’s a bit misleading to say that ‘survival is paramount – and that every strategy serves that goal’: survival plays a crucial role in evolution, but it is not the ultimate goal. As emphasised by Charles Darwin’s theory of…
Value Space and “Good” Human Values Human values can be conceptualised as occupying regions in a vast objective multidimensional “value space.” These regions reflect preferences for cooperation, survival, flourishing, and minimising harm, among other positive traits. If TAI / SI can approximate the subset of human values deemed “good” (e.g., compassion, fairness, cooperation), while avoiding…
Note: this is by no means exhaustive. it’s focus is on western philosophy. I’ll expand and categorise more later… The earliest precursors to indirect normativity can be traced back to early philosophical discussions on how to ground moral decision-making in processes or frameworks rather than specific, static directives. While Nick Bostrom’s work on indirect normativity…
Kristian Rönn discusses his amazing book The Darwinian Trap – Macroeconomics & Game theoretic Imagineering: avoiding the minefields of the tragedy of the common in the hope that we can stave off the onslaught of Darwinian Demons, and ultimately find some nash equilibrium of fairness for all humanity, sentient AI an non-human animals. The book…
Will AI Ignore, Preserve or Improve on Our Values? Imagine a future where advanced AIs follow every instruction flawlessly – but humanity feels strangely adrift. Our tools are obedient, but the ‘soul’ of our civilisation feels absent. This is the hidden danger of V-risk – value erosion. In this post I explore what I broadly…
The concept of a non-arbitrary objective value space offers a compelling framework for understanding how to navigate to more valuable worlds. This framework presupposes the existence of stance-independent normative truths – metaphysical/ontological, epistemic, scientific & ethical principles that rational agents will converge upon given sufficient cognitive sophistication [1]. Here I will mostly talk in relation…
Is self-interest by necessity the default motivation for all agents?Will Superintelligence necessarily become a narcissistic utility monster? No, self-interest is not necessarily the default motivation of all agents. While self-interest is a common and often foundational drive in many agents (biological or artificial), other motivations can and do arise, either naturally or through design, depending…
In AI safety, Control and Motivation are the two primary strategies used to prevent advanced AI systems from causing harm. While control focuses on external constraints, motivation addresses the internal goals and values of the AI. Most researchers, including those at Anthropic and the Future of Life Institute, suggest a hybrid approach: using strict capability…
In this interview Jeff Sebo discusses the ethical implications of artificial intelligence and why we must take the possibility of AI sentience seriously now. He explores challenges in measuring moral significance, the risks of dismissing AI as mere tools, and strategies to mitigate suffering in artificial systems. Drawing on themes from the paper ‘Taking AI…