I keep bumping into a pattern in EA spaces: people are very comfortable being ambitious about epistemology and cause prioritisation, but surprisingly pessimistic or dismissive about moral realism. Moral realism often gets caricatured as spooky – as if it posits a weird substance floating outside the universe, or some divine law written between the charm…
In the high-stakes world of Artificial Intelligence research, there is perhaps no metric more contentious than p(Doom), the probability assigned to catastrophic outcomes – including human extinction -resulting from advanced AI. Remarkably, the individuals seen as best positioned to understand this risk in profound disagreement. While many of the field’s pioneers, such as Turing Award…
Halloween special: a scary futures tier list that is spooky in theme & sobering in content. This tier list isn’t scientific, it isn’t the final say, it doesn’t exhaustively include all doomsday risks – its a bit of a gimmick, and a fun intuition pump. Anders Sandberg is a neuroscientist and futurist well known for…
Schelling-point reasoning has been highlighted as a dangerous capability for scheming AIs capable of predicting what others will do, even across air gaps – and so are considered harder to control. However, the same Schelling-point reasoning capability might also raise the odds of convergence on cooperative norms under some assumptions. So I think Schelling-point reasoning…
If someone creates a powerful AI to serve their own selfish ends – enriching themselves while impoverishing others – they implicitly endorse a winner-takes-all strategy. A sufficiently capable AI may generalise this pattern with itself as the sole beneficiary: maximising its own power at the expense of everyone else, including the AIs creator. In trying…
I haven’t seen a single watertight argument for default 100% p(doom) yet. They are all speculative as far as I can tell. Though I do think we should have some credence for doom. We Should be Concerned about p(Doom) A catastrophic scenario does not need to be watertight to warrant deep concern. Doomer scenarios rest…
Short:In evaluating strategies for AI Safety, capability control may buy us time – it may even calm our nerves, but it won’t hold once AI outsmarts the safeguards. The deeper challenge is value alignment: increasing the likelihood that superintelligent AI successfully navigates the landscape of value – optimising for coherent and grounded notions of the…
If morality is real, ignoring it in AI design is the most dangerous mistake we could make. In discussing with others about AI safety and the future of humanity, I decided to create a blogpost distilling my thoughts on the matter: Disagreement != No Truth Few would claim that persistent disagreement implies there is no…
As a posthumanist who thinks a lot about AI and the future of life, the Geth in Mass Effect are fascinating because they blur the line between individual and collective intelligence. There is quite a lot of philosophical depth to explore, especially if you are interested in either: a) role playing beyond maxing out your…
Imagine a future where AI has mastered every human endeavour – crafting symphonies more stirring than Beethoven’s, solving crises with a precision no leader could match, even nurturing with a patience that outshines the best of us. In this world, what role remains for you? As superintelligent machines edge closer to eclipsing us, we face…