We shouldn’t just design AI to behave nicely in our human sandbox in case the sandbox turns out to be part of a much larger galactic metropolis. In this case we’d better make sure the AI doesn’t start flinging galactic sand in anyone’s eyes. We may at some stage encounter technologically mature civilisations far more…
No one in their right mind can deny the struggle to survive animates natural selection. Though it’s a bit misleading to say that ‘survival is paramount – and that every strategy serves that goal’: survival plays a crucial role in evolution, but it is not the ultimate goal. As emphasised by Charles Darwin’s theory of…
Note: this is by no means exhaustive. it’s focus is on western philosophy. I’ll expand and categorise more later… The earliest precursors to indirect normativity can be traced back to early philosophical discussions on how to ground moral decision-making in processes or frameworks rather than specific, static directives. While Nick Bostrom’s work on indirect normativity…
Implications for AI Safety and Indirect Normativity The challenge of understanding value space has moved from a philosophical curiosity to an urgent technical problem. As we approach the development of advanced artificial intelligence, the structure of value and our ability to specify it has become perhaps the most critical challenge facing humanity. This challenge assumes…
Will AI Ignore, Preserve or Improve on Our Values? Imagine a future where advanced AIs follow every instruction flawlessly – but humanity feels strangely adrift. Our tools are obedient, but the ‘soul’ of our civilisation feels absent. This is the hidden danger of V-risk – value erosion. In this post I explore what I broadly…
Some people want to infuse AI with the codification of our existing moral judgements. CEV takes the idea a bit further. In short, CEV is a conceptual approach to aligning superintelligent AI by designing it to act according to an idealised version of what humans would want if they had more knowledge, were faster thinkers,…
The concept of a non-arbitrary objective value space offers a compelling framework for understanding how to navigate to more valuable worlds. This framework presupposes the existence of stance-independent normative truths – metaphysical/ontological, epistemic, scientific & ethical principles that rational agents will converge upon given sufficient cognitive sophistication [1]. Here I will mostly talk in relation…
Is self-interest by necessity the default motivation for all agents?Will Superintelligence necessarily become a narcissistic utility monster? No, self-interest is not necessarily the default motivation of all agents. While self-interest is a common and often foundational drive in many agents (biological or artificial), other motivations can and do arise, either naturally or through design, depending…
In AI safety, Control and Motivation are the two primary strategies used to prevent advanced AI systems from causing harm. While control focuses on external constraints, motivation addresses the internal goals and values of the AI. Most researchers, including those at Anthropic and the Future of Life Institute, suggest a hybrid approach: using strict capability…
Part 1 in an interview with Eric Sampson on Moral Realism. Note AI is referenced, but the bulk of the questions about AI are covered in part 2. Chapters 0:00 Intro 3:03 Why is moral realism attractive to AI Safety Research? 2:02 Background, & Personal reason to choose moral realism 4:00 Describing moral realism –…