In his 2011 paper, Complex Value Systems are Required to Realize Valuable Futures , Eliezer Yudkowsky posits that aligning artificial general intelligence (AGI) or artificial superintelligence (ASI) with human values necessitates embedding the complex intricacies of human ethics into AI systems. He warns against oversimplification, suggesting that without a comprehensive inheritance of human values, AGI…
Norm-sensitive cooperation might beat brute-force optimisation when aligning AI with the complex layers of human value – and possibly cosmic ones too. Nick Bostrom suggests that AI systems designed with humility and a cooperative orientation are more likely to navigate the complex web of human and potentially cosmic norms than those driven by rigid utility…
In short: Satiable values have diminishing returns with more resources, while insatiable values always want more, potentially leading to risky AI behaviour – it seems likely that AI with insatiable values could make existential trades, like gambling the world for a chance to double resources. Therefore potentially design AI with satiable values to ensure stability,…
Value Space and “Good” Human Values Human values can be conceptualised as occupying regions in a vast objective multidimensional “value space.” These regions reflect preferences for cooperation, survival, flourishing, and minimising harm, among other positive traits. If TAI / SI can approximate the subset of human values deemed “good” (e.g., compassion, fairness, cooperation), while avoiding…
Implications for AI Safety and Indirect Normativity The challenge of understanding value space has moved from a philosophical curiosity to an urgent technical problem. As we approach the development of advanced artificial intelligence, the structure of value and our ability to specify it has become perhaps the most critical challenge facing humanity. This challenge assumes…
Will AI Ignore, Preserve or Improve on our Values? Imagine a future where advanced AIs follow every instruction flawlessly – but humanity feels strangely adrift. Our tools are obedient, but the ‘soul’ of our civilisation feels absent. This is the hidden danger of V-risk—value erosion. In this post I explore what I explore what I…
The concept of a non-arbitrary objective value space offers a compelling framework for understanding how to navigate to more valuable worlds. This framework presupposes the existence of stance-independent normative truths – metaphysical/ontological, epistemic, scientific & ethical principles that rational agents will converge upon given sufficient cognitive sophistication [1]. Here I will mostly talk in relation…
The meaning of homo sapien means ‘wise man’ – given the many current human–caused precarious states of affairs, this self-description seems a bit of a reach. What are values? Values are beliefs or ideals that help determine what is important and how one should act. They can be thought of as a framework that guides…
Anders Sandberg discusses ideas & values and where we get them from, mindsets for progress, and that we are living in a unique era of technological change but also, importantly we are aware that we are living in an era of great change. Is there a direction in ethics? Is morality real? If so, how…
One can think of Existential Risk as a subcategory of a Global Catastrophic Risk – while GCR’s are really bad, civilization has the potential to recover from such a global catastrophic disaster. An existential Risk is one in which there is no chance of recoverability. An example of the sort of disaster that fits the…
