Schelling-point reasoning has been highlighted as a dangerous capability for scheming AIs capable of predicting what others will do, even across air gaps – and so are considered harder to control. However, the same Schelling-point reasoning capability might also raise the odds of convergence on cooperative norms under some assumptions. So I think Schelling-point reasoning…
If you train or steer a powerful AI to serve selfish ends – enriching yourself while impoverishing others – you implicitly endorse a winner-takes-all strategy. A sufficiently capable AI may generalise this pattern, but with itself as the sole beneficiary: maximising its own power at the expense of everyone, including you. In trying to weaponise…
Cunning + luck ≈> brute-force intelligence – (cascading caveats ensuing).. In a densely populated landscape of naively or sub-adequately regulated selection pressures, raw cunning combined with moderate intelligence and high risk-tolerance will often outperform high IQ (esp. in the short to medium term). This holds so long as peak cunning matches or exceeds the higher…
Blue Dot Impact recently published a detailed overview of seven key strategies in AI safety – ranging from international cooperation to domestic race-winning and societal resilience. It’s the result of 100+ hours of analysis, interviews, and synthesis by Adam Jones, and it’s a solid strategic landscape mapping effort. I was made aware of this via…
In his 2011 paper, Complex Value Systems are Required to Realize Valuable Futures , Eliezer Yudkowsky posits that aligning artificial general intelligence (AGI) or artificial superintelligence (ASI) with human values necessitates embedding the complex intricacies of human ethics into AI systems. He warns against oversimplification, suggesting that without a comprehensive inheritance of human values, AGI…
Surviving an intelligence explosion is just Level One. Another layer to the Great Filter might be whether our civilisation, or a greedy AI singleton can grow up enough not to become a cosmic jerk. You might brute-force your way to technological maturity, but that doesn’t mean you get a seat at the galactic grown-up table….
I just finished Mo Gawdat’s book Scarey Smart. The overview says: By 2049 AI will be a billion times more intelligent than humans. Scary Smart explains how to fix the current trajectory now, to make sure that the AI of the future can preserve our species. This book offers a blueprint, pointing the way to…
What if superintelligent AI didn’t care about humans? Does intelligence necessitate care? Probably not.While I find it hard to imagine care in the absence of cognitive capacity, I can’t see anything that guarantees all cognitively apt agents will harbour care. How, if at all possible, can we from our current position assess the likelihood that…
In AI safety, direct specification refers to the attempt to program specific goals, rules, or behaviours directly into an AI system. AI trained to optimise values may over-optimise those we specify at the expense of real value we don’t specify. Political or market dynamics incentivise racing to powerful AI – the race dynamics come with…
In interesting article I read today was ‘Meta’s AI guru LeCun: Most of today’s AI approaches will never lead to true intelligence‘ by Tiernan Ray. Though I think the title is a bit misleading with it’s reference to ‘true intelligence’. The article uses ‘human level intelligence’ as it’s framing, though the space of possible intelligence…
