When the machines outthink us, will they outmaneuver us? Barrat’s latest is a wake-up call we can’t afford to ignore. Having had the privilege of previewing James Barrat’s latest work, The Intelligence Explosion (Amazon link, Google Book link), I found it to be a compelling and thought-provoking exploration of the rapidly evolving landscape of artificial…
This post will be updated over the coming weeks. In his 2011 paper, Complex Value Systems are Required to Realize Valuable Futures, Eliezer Yudkowsky posits that aligning artificial general intelligence (AGI) with human values necessitates embedding the complex intricacies of human ethics into AI systems. He warns against oversimplification, suggesting that without a comprehensive inheritance…
Norm-sensitive cooperation might beat brute-force optimisation when aligning AI with the complex layers of human value – and possibly cosmic ones too. Nick Bostrom suggests that AI systems designed with humility and a cooperative orientation are more likely to navigate the complex web of human and potentially cosmic norms than those driven by rigid utility…
Surviving an intelligence explosion is just Level One. Another layer to the Great Filter might be whether our civilisation, or a greedy AI singleton can grow up enough not to become a cosmic jerk. You might brute-force your way to technological maturity, but that doesn’t mean you get a seat at the galactic grown-up table….
I just finished Mo Gawdat’s book Scarey Smart. The overview says: By 2049 AI will be a billion times more intelligent than humans. Scary Smart explains how to fix the current trajectory now, to make sure that the AI of the future can preserve our species. This book offers a blueprint, pointing the way to…
Nick Bostrom: Superintelligence Could Be a Year Away—or Not Back in 2012, when I first chatted with Nick Bostrom about AI risk, the topic was a niche blip—dung beetle mating habits had more academic ink. Fast forward to 2025, post his 2014 game-changer ‘Superintelligence: Paths, Dangers, Strategies’, and AI’s sprinting ahead. So, how close are…
Nick Bostrom dropped a bombshell that cuts through some of the haze around AI doom-and-gloom: “I think ultimately this transition to the superintelligence era is one we should do.” That’s right—should. Not “might” or “could,” but a firm nudge toward a future where humanity doesn’t just survive AI but thrives with it. Even more striking?…
In short: Satiable values have diminishing returns with more resources, while insatiable values always want more, potentially leading to risky AI behaviour – it seems likely that AI with insatiable values could make existential trades, like gambling the world for a chance to double resources. Therefore potentially design AI with satiable values to ensure stability,…
Super excited that I’ll be interviewing Nick Bostrom again in a few days – last time was in 2012 (on XRisks of AI and the Simulation Argument). This time we will cover topics that range from Superintelligence to Deep Utopia. Since Nick Bostrom wrote Superintelligence, AI has surged from theoretical speculation to powerful, world-shaping reality….
About “I argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood, i.e. of AI systems with their own interests and moral significance, is no longer an issue only for sci-fi or the…
