More precisely, Deepmind’s Gemini AI achieved gold standard (it didn’t technically win) at the International Math Olympiad. AI has made significant gains in capability over the past few years. Could AI be a better moral reasoner than us, and how would we know? This is a genuinely important question that highlights a tension in how…
I recently read about Critical Realism (CR), it’s interesting – I like a lot of it, though I’m not sure I fully buy the layered picture with latent causal structure at the bottom, the “actual” layer floating above that, and the “empirical” layer on top. So CR back against a certain kind of reductive physicalism…
I keep bumping into a pattern in EA spaces: people are very comfortable being ambitious about epistemology and cause prioritisation, but surprisingly pessimistic or dismissive about moral realism, or seem attracted to moral anti-realism. Moral realism often gets caricatured as spooky – as if it posits a weird substance floating outside the universe, or some…
AI doesn’t need a moustache-twirling villain to go wrong – it just needs the wrong metaethics in an unforgiving game. In this post I argue that: (1) many human moral norms are shaped by competitive, “Molochian” pressures; (2) if we give AI a purely constructivist / anti-realist metaethic, it will inherit and amplify those pressures;…
In the high-stakes world of Artificial Intelligence research, there is perhaps no metric more contentious than p(Doom), the probability assigned to catastrophic outcomes – including human extinction -resulting from advanced AI. Remarkably, the individuals seen as best positioned to understand this risk in profound disagreement. While many of the field’s pioneers, such as Turing Award…
Schelling-point reasoning has been highlighted as a dangerous capability for scheming AIs capable of predicting what others will do, even across air gaps – and so are considered harder to control. However, the same Schelling-point reasoning capability might also raise the odds of convergence on cooperative norms under some assumptions. So I think Schelling-point reasoning…
If someone creates a powerful AI to serve their own selfish ends – enriching themselves while impoverishing others – they implicitly endorse a winner-takes-all strategy. A sufficiently capable AI may generalise this pattern with itself as the sole beneficiary: maximising its own power at the expense of everyone else, including the AIs creator. In trying…
I haven’t seen a single watertight argument for default 100% p(doom) yet. They are all speculative as far as I can tell. Though I do think we should have some credence for doom. We Should be Concerned about p(Doom) A catastrophic scenario does not need to be watertight to warrant deep concern. Doomer scenarios rest…
Cunning + luck ≈> brute-force intelligence – (cascading caveats ensuing).. In a densely populated landscape of naively or sub-adequately regulated selection pressures, raw cunning combined with moderate intelligence and high risk-tolerance will often outperform high IQ (esp. in the short to medium term). This holds so long as peak cunning matches or exceeds the higher…
Blue Dot Impact recently published a detailed overview of seven key strategies in AI safety – ranging from international cooperation to domestic race-winning and societal resilience. It’s the result of 100+ hours of analysis, interviews, and synthesis by Adam Jones, and it’s a solid strategic landscape mapping effort. I was made aware of this via…