Blue Dot Impact recently published a detailed overview of seven key strategies in AI safety – ranging from international cooperation to domestic race-winning and societal resilience. It’s the result of 100+ hours of analysis, interviews, and synthesis by Adam Jones, and it’s a solid strategic landscape mapping effort. I was made aware of this via…
TL;DR Superintelligence has strong instrumental reasons to adopt a generally cooperative posture if it deems it likely that mature civilisations may prefer cooperation (a universal leviathan), run deep audits on it’s history and psychology and penalise the superintelligence if there are signatures of defection found, making cooperation (and perhaps moral reliability) a long-term survival strategy,…
Are you skeptical that controlling AI is enough? We have to make it care about the right things. As AI systems become more capable, we face a question more foundational than how to contain them: what values should they be aligned to? Upstream value alignment challenges the conventional reliance on “capability control” – tripwires, sandboxing,…
In his 2011 paper, Complex Value Systems are Required to Realize Valuable Futures , Eliezer Yudkowsky posits that aligning artificial general intelligence (AGI) or artificial superintelligence (ASI) with human values necessitates embedding the complex intricacies of human ethics into AI systems. He warns against oversimplification, suggesting that without a comprehensive inheritance of human values, AGI…
Surviving an intelligence explosion is just Level One. Another layer to the Great Filter might be whether our civilisation, or a greedy AI singleton can grow up enough not to become a cosmic jerk. You might brute-force your way to technological maturity, but that doesn’t mean you get a seat at the galactic grown-ups table….
In short: Satiable values have diminishing returns with more resources, while insatiable values always want more, potentially leading to risky AI behaviour – it seems likely that AI with insatiable values could make existential trades, like gambling the world for a chance to double resources. Therefore potentially design AI with satiable values to ensure stability,…
Firstly, values are important because they are the fundamental principles that guide decisions and actions in humans. They shape our understanding of right and wrong, and they influence how we interact with the world around us. In the context of AI, values are particularly important because they determine how AI systems will behave, what goals…
Implications for AI Safety and Indirect Normativity The challenge of understanding value space has moved from a philosophical curiosity to an urgent technical problem. As we approach the development of advanced artificial intelligence, the structure of value and our ability to specify it has become perhaps the most critical challenge facing humanity. This challenge assumes…
Capability Control: The Tool or Slave Model Capability control focuses on limiting what an AI can do, rather than shaping why it does it. The idea is to ensure AI systems behave predictably and stay within strict bounds – like a powerful but obedient tool. This includes: In this model, the AI is treated less…
This is an attempt to formally introduce the concept of value risk (v-risk or VRisk). This is not an attempt at repackaging well known established risk categories like x-risk and s-risk, it’s an attempt to carve out axis of concerns about value choice, hopefully to help increase clarity and weight to what might otherwise sound…
