Surviving an intelligence explosion may just one of the middle layers in the Great Filter. Another layer to the Great Filter might be whether our civilisation, or a greedy AI singleton can grow up enough not to become a cosmic jerk. You might brute-force your way to technological maturity, but that doesn’t mean you get…
I just finished Mo Gawdat’s book Scarey Smart. The overview says: By 2049 AI will be a billion times more intelligent than humans. Scary Smart explains how to fix the current trajectory now, to make sure that the AI of the future can preserve our species. This book offers a blueprint, pointing the way to…
Are we living in the final moments of the slow era? Nick Bostrom: Superintelligence Could Be a Year Away – or Not Back in 2012, when I first chatted with Nick Bostrom, AI risk was a fringe academic curiosity, rarely surfacing in mainstream discourse. That changed in 2014 with Nick Bostrom’s Superintelligence, which established a…
Nick Bostrom dropped a bombshell that cuts through some of the haze around AI doom-and-gloom: “I think ultimately this transition to the superintelligence era is one we should do.” That’s right—should. Not “might” or “could,” but a firm nudge toward a future where humanity doesn’t just survive AI but thrives with it. Even more striking?…
In short: Satiable values have diminishing returns with more resources, while insatiable values always want more, potentially leading to risky AI behaviour – it seems likely that AI with insatiable values could make existential trades, like gambling the world for a chance to double resources. Therefore potentially design AI with satiable values to ensure stability,…
The interview has now been released! We covered a lot of stuff which at the time of writing hasn’t been covered elsewhere in detail. Super excited that I’ll be interviewing Nick Bostrom again in a few days – last time was in 2012 (on XRisks of AI and the Simulation Argument). This time we will…
“I argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood, i.e. of AI systems with their own interests and moral significance, is no longer an issue only for sci-fi or the distant…
Nick Bostrom addresses concerns about biased influence on the extrapolation base in his discussions on indirect normativity (IN – i.e., an approach where the AI deduces ideal values or states of affairs rather than running with current values and states of affairs) esp. coherent extrapolated volition (CEV) and value alignment in his magnum opus Superintelligence….
Do LLMs have metacognition? What happens when machines begin to “think about thinking”? At Future Day 2025, we hosted an in-depth discussion on metacognition in large language models (LLMs) – with insights from researcher Shun Yoshizawa, supervised by neuroscientist Dr Ken Mogi at Sony Computer Science Laboratories. Together, we explored the current state of metacognitive…
A panel between James Hughes, James Newton-Thomas, Lev Lafayette and Adam Ford (chair) at Future Day 2025. Video of the panel – ‘AI Automation: Human Wage Crash Panel’: About If AI drives human wages to the ground, should everything else crash too? Venture capitalist Marc Andreessen envisions a future where AI-induced wage collapse leads to…