Radical Interpretability

When Interpretability Becomes Mind-Reading

Seatbelt Interpretability

Most AI interpretability research asks a near-term engineering question: can we understand enough of a model’s internal processing to explain, predict or control important behaviour? Why did it refuse this request? Which internal features mattered? Is it representing a hidden objective? Can we tell whether a safety mechanism is actually doing what we think it is?

Call this seatbelt interpretability: understanding enough of the machinery to make the system safer.

Engineers want answers to questions like:  Why did the model recommend this video?… flag this message?…tag this as X? What feature triggered that refusal? Where is the bug? Is the model lying?  Is it hiding a goal?

Radical Interpretability

I define radical interpretability for a much stronger capability observing the physical or computational state and information flow of a cognitive system and recovering, with high fidelity, the beliefs, goals, plans, uncertainties and other cognitive variables doing causal work inside it.¹

It is mind-reading without mysticism. Instead of inferring a mind from what it says or does, one can inspect the machinery while it runs.

Ideal interpretability probably the upper limit – something approaching complete, reliable and near-real-time access to every cognitively relevant state. Radical interpretability may never reach that ideal, and may never need to. Perhaps it only needs to become good enough that strategically important beliefs, motives and plans can no longer be reliably hidden from a sufficiently capable observer with adequate access.

If that endpoint arrives, it will be far more than a tool for aligning AI models in labs. It will shape how mature civilisations bargain, cooperate, punish cheating, and even communicate².

This post explains that idea in plain terms, then looks at the kinds of futures it points to – and discusses some caveats that may help stop it being misinterpreted as a fantasy novel festooned in sci-fi regalia.

That includes:

• what the agent believes,
• what it wants,
• what it is about to do,
• and what would change its mind.

It is not reading thoughts by watching facial expressions, mystical telepathy (ESP) or mind-reading by vibes.

It is closer to what a skilled programmer does when they read code, or better yet, add traces, breakpoints and watches and understands what is going on as the code executes – except the code is a vast neural network of numbers, and the reader has tools that translate patterns into concepts.

Why should minds be interpretable at all?

Radical interpretability rests on the assumption that cognition has discoverable causal structure. A brain or artificial mind cannot merely be an arbitrary cloud of patterns in which arbitrary internal states somehow produce useful behaviour. There have to be regularities linking the way the system is organised, the information flowing through it, and what it eventually does.

There are several reasons to expect this.

First, minds are physical systems. Whatever their substrate – neurons, silicon, photonics or something stranger – one physical state produces another according to regularities in the underlying world. Intelligence does not get an exemption from causality merely because the machinery is complicated.

Second, a system that behaves intelligently across many different situations must itself exploit regularities. A mind that recognises a cow it has never seen before cannot simply contain an unrelated trick for every possible cow.³ Somewhere in its computation it must capture features that generalise. The representation may be distributed, dynamic and extraordinarily alien to us, but successful generalisation is itself evidence that some compressible structure exists.

Third, optimisation takes place under constraints. Memory, energy, communication bandwidth, latency and reliability all have costs. Reusing representations, computations and control structures can be cheaper than reinventing them for every situation. We should therefore expect recurring mechanisms and abstractions even when their implementation is messy.

The human genome offers a useful analogy, provided we do not push it too far. Reading the genome did not reveal a tidy dictionary in which one gene corresponds to one visible trait. Gene expression depends on networks, development and environment. Yet the apparent complexity did not turn biology into arbitrary chaos. We can still discover systematic relations between molecular structure, regulation and phenotype. I think human cognition may turn out similarly: not a clean codebook, but a complicated system containing enough regular causal structure to support progressively better explanation.

The same qualification applies to neural networks. We should not expect one neuron or one weight to mean “honesty”, “Paris” or “bring that beer over here”. Different implementations can perform the same computation, and information can be distributed across many components. The right objects of interpretation may instead be causal pathways, representations, dynamical structures and functional invariants.

That distinction actually makes radical interpretability more plausible. The goal is not to attach an English label to every atom or parameter. It is to recover the higher-level variables that matter: what information the system has, which possibilities it is considering, what it values, what it expects to happen and which computations lead it towards one action rather than another.

And the civilisation attempting this may not resemble us. We are trying to reverse-engineer minds with primitive instruments and a young science of cognition. A civilisation with thousands or millions of additional years of scientific and engineering progress might be able to record cognitive systems at extraordinary resolution, perturb them experimentally, compare copies, run detailed simulations and use ultraintelligent scientists to discover the appropriate abstractions.

Artificial minds could make the problem easier still. Mature civilisations may design their cognitive architectures themselves. Cooperative systems might contain dedicated audit interfaces, preserve interpretable developmental records, or expose selected internal variables by design. Radical interpretability would then cease to be purely a problem of reverse-engineering an alien black box.

None of this implies omniscience. Some computations may resist shortcuts: knowing the machinery may not let you predict every future thought without effectively running it. An adversarial mind may deliberately encrypt or obscure parts of its cognition. And different systems may implement the same cognitive function in radically different ways. Radical interpretability therefore need not mean “know everything instantly”. It means that minds contain enough discoverable causal structure for sufficiently advanced observers to recover the cognitive facts that matter.

Interpreting Politicians

If radical interpretability is interrogating intentions at their starting point (causal starting point) – in the case of the politician it’s like having access to their private memos instead of just relying on hearing the speech – except the memos are patterns in a brain, an artificial neural network, or whatever future architecture carries thought. 

Like a super advanced dashboard with all sorts of graphs, scorecards and scatterplots reporting the health of a business, think of RI a dashboard for an agent that can reason, bargain, threaten, deceive, cooperate, and plan years ahead.

Why the idea is not crazy

We already have crude hints that internal cognitive states are decodable. Researchers have reconstructed aspects of the semantic content of perceived and imagined language from human fMRI recordings, after training decoders on individual participants. Other experiments have reconstructed aspects of visual experience from patterns of brain activity. These systems are nowhere near radical interpretability – and some require substantial cooperation from the person being decoded – but they establish the more modest point that internal cognitive states contain recoverable structure.⁴

Also mechanistic interpretability is getting more precise through better tools and methods like linear probes, sparse autoencoders, circuit analysis, and automated evaluation.

Furthermore, singular learning theory (SLT) is interesting here partly because it warns against an overly simple picture of interpretability. Neural networks can contain many different parameter configurations that implement equivalent or near-equivalent functions. The exact weights may therefore be the wrong level at which to look for meaning. What matters may instead be the geometry of learned functions, recurring computational structures, causal roles and other invariants across different implementations.⁵ It’s worth emphasising that law-like regularity does not require a unique implementation.

If you push that direction far enough, you get a future where a mature AI can do what today’s interpretability tools can only hint at:

• trace a decision back to the internal steps that produced it,
  
• flag hidden motives,
  
• spot deception as it forms,
  
• and check whether a promise is backed by a real commitment.
  

That last point – commitment to promises – matters. In human life, promises are often cheap. In a world with radical interpretability, a promise could be inspected like a signed contract.

Game Theory: what happens when minds are readable?

Game theory often assumes you can’t see what the other players want, and they can’t see what you want. Many strategic problems become harder when players possess private information: what they want, what they know, whether a threat is genuine, whether a promise will survive temptation, or whether apparently peaceful behaviour conceals preparations for attack.

Diplomacy and war cannot be reduced to uncertainty about intentions, but that uncertainty can intensify security dilemmas. A civilisation that cannot distinguish a peaceful neighbour from a predator may arm itself, refuse mutually beneficial agreements or even consider striking first.

Radical interpretability could remove part of that uncertainty – especially where access is reciprocal and the interpretation itself can be trusted.

That is why diplomacy sometimes fails, why wars start, and why fraud works. If you cannot tell whether the other party is honest, you must hedge. You build weapons just in case, you strike first just in case⁶, you refuse deals just in case and so on…

Radical interpretability clears up a lot of that uncertainty.

Example: the handshake you can audit

Imagine two powerful AIs meet in space. Each offers a treaty: “I won’t attack you if you won’t attack me.”

In our world, those words are a thin layer of paint. The real question is: does the treaty bind their future selves?

With radical interpretability, a third party (or each side) might inspect internal mechanisms that enforce the treaty. You could look for something like:

• a rule that triggers penalties if the agent violates the treaty,
• a tripwire that alerts others,
• a constraint that blocks certain kinds of action even under temptation.

If those mechanisms are real, trust becomes less of a leap. Cooperation becomes cheaper.

The future may contain far less naive optimists waiting to be mugged by cruel Machiavellians.

Of course, reading a commitment is not enough if the agent can quietly remove it tomorrow. A mature audit would also need to inspect the mechanisms governing self-modification, delegation and the creation of successor agents. The stronger possibility is therefore ongoing auditability: not merely seeing that an agent currently intends to cooperate, but checking whether the processes that could change that intention are themselves constrained or observable.

The first caveat: competence doesn’t guarantee goodness

You’ve probably come across the orthogonality thesis⁷. A mature superintelligence can be brilliant and still be awful and it’s brilliance may just tell it how to win, not what to want.

So when you imagine the moral arc of the long-term future, do not assume that a more capable mind automatically lands in a benevolent region of the value landscape⁸. It might land in a region that is stable, powerful, and horrifying.

Interpretability can reveal goals. It cannot, by itself, make goals good.

The second caveat: full transparency may be unstable

Total mind-reading sounds like a recipe for peace. In scenarios with an ounce of adversity, it may also be a recipe for paranoia.

If you can read my mind, you can also:

• learn my weaknesses,
• steal my tricks,
• predict my defences,
• and strike before I can react.

So radical interpretability creates an arms race where one side develops better mind-reading tools. The other side develops better mind-shielding tools (think of encryption, but for thoughts).

A stable future may not converge on a state where everyone reads everyone. It may converge on selective transparency.

In selective transparency, you do not open your entire mind – you prove what matters. You prove you are safe to cooperate with. You prove you will obey the treaty. You prove you are not hiding a weapon. You keep the rest private. This kind of proof may look less like mind-reading and more like auditing. It may even resemble “show me you comply” rather than “show me everything you are.”

Anders Sandberg anticipated a version of this problem in his work on whole-brain emulations.⁹ A software mind could, in principle, expose its internal state to whoever controls the hardware on which it runs. Sandberg therefore treats mental transparency not merely as a scientific opportunity but as a profound security and privacy problem, and considers technological protections such as encrypted computation and tamper-resistant hardware.

This suggests an important distinction: radical interpretability may be technologically possible without universal transparency becoming socially acceptable. Mature civilisations might simultaneously possess extraordinary mind-reading and extraordinary mind-shielding technology.

The third caveat: there may be no single “best” morality

People often imagine a single peak in the landscape of value: the moral truth, waiting at the top like a flag on Everest. A multi-agent universe may be far messier.

Even if there are objective moral truths, the path to them may not be unique, and there may be many comparably high peaks in the landscape of value. Civilisations may settle into different stable patterns depending on:

• who arrived first,
• who holds power,
• which kinds of commitments are possible,
• which kinds of verification are cheap,
• and how much diversity they tolerate.

So the far future may contain not one moral city, but a federation of cities with shared laws at the border – those shared laws will likely be thin and serious: rules against extreme harm, reckless experiments, and predatory conquest. Inside the borders, local norms may differ.¹⁰

That is not moral collapse. It is what stability often looks like when power is large and values are not identical.

Radical interpretability meets moral realism

Lets add another hot claim to the embers: if radical interpretability can decode minds, it might decode experience.

This possibility depends on several claims. First, conscious experience must depend systematically on physical or computational states. Second, positive and negative valence – how good or bad an experience feels – must be a real feature of those states that can be measured. Third, if we want to move from measurement to morality, valence must have objective moral significance.¹¹

It might let us detect suffering and bliss directly from brain states – or from whatever physical states play the same role in synthetic minds. It’s hard to imagine a future with moral bookkeeping, where every unit of pain and joy is counted with receipts. But I think it’s worth trying.

If something like naturalistic valence realism is true, it could afford traction to the empirical side of ethics. Instead of relying mainly on behaviour and self-report, a mature civilisation might estimate suffering and wellbeing from the processes producing experience itself.

In the ideal case, welfare accounting might even come with something resembling accounting beyond tracking reports of suffering – providing receipts for measured units of suffering (or at least estimates of the units).

That still leaves formidable problems. How do we compare valence across a human, an octopus, a digital mind and an alien? Is twice the measured intensity really twice the moral importance? Are all dimensions of value reducible to valence? Radical interpretability could make these questions empirically tractable without guaranteeing that they have simple answers.

Game theory would then have a different job. I don’t think it would not manufacture moral value out of whatever equilibrium happens to survive. It would be useful in helping determine which morally valuable arrangements are stable, reachable and resistant to defection.

I am hopeful that this is possible – after all, valence realism (or valence naturalism) suggests that everything there is to how we feel comes down to the regularities of physics that everything else we can measure is beholden to.¹² However, we can’t know how far we can take valence reading yet. So, source normativity from valence, and let game theory do feasibility and convergence work. If the payoffs are welfare payoffs, and welfare is stance-independently good or bad, then the payoffs arrive pre-loaded with normativity and the game theory is telling us which arrangements among valenced agents are stable and reachable. That is a clean division of labour in that valence realism is doing the axiology, and game theory (perhaps something like evolutionary game theory) could be doing the dynamics. It also explains why convergence would be evidentially interesting without having to claim that stability constitutes value.

What radical interpretability could genuinely improve

It could make hidden suffering harder to ignore by allowing the comparison of what a system says about its experience with independent evidence from its internal organisation – especially if future theories of consciousness make testable predictions about which structures or dynamics support experience.

It could help us find suffering in places we may currently overlook.

What it might not solve

Even with perfect access to internal states hard problems remain, and without testing we don’t know how far we could take it.

• Scaling: how do you compare “my pain” to “your pain” across very different minds?
• Calibration: what counts as twice the suffering?
• Gaming: if we reward a welfare score, agents may learn to hack the score instead of improving welfare.
• Norms: even perfect measurement of experience does not, by itself, settle every moral question.

Radical interpretability could support auditable, evidence-based estimates of welfare, with clear uncertainty, but this may not come with an ideal hedonic valence ledger.

What a mature ethics programme would do with all this

Imagine a superintelligence whose ethical development does not end with whatever values its creators managed to write down in the twenty-first century. It continues tracking reality, correcting mistakes and reflecting over increasingly long horizons.

Eventually, such a superintelligence must ask what ethics looks like when the relevant society is no longer humanity alone, but a universe containing many mature intelligences. What happens when those intelligences meet? Which arrangements preserve value? Which incentives produce cooperation rather than predation? What prevents a disagreement from becoming a war fought with astronomical resources?

Radical interpretability matters here because it changes the information available to the players.

In a universe of many powerful agents, a mature programme will focus on stability:

• How do cooperators punish cheaters without starting wars?
• How do you prevent arms races from eating everything?
• What kinds of commitments can be verified cheaply?
• What must be transparent, and what must remain private?
• Which moral rules still make sense when the cost of conflict is astronomical?

This is where radical interpretability matters most.

It can lower the cost of trust, reduce accidental conflict, make commitments real, expose predation early and give moral arguments a firmer grip on facts about experience.

But it also creates new dangers – especially if mind-reading is one-sided, or if it destroys privacy so thoroughly that everyone starts building mental armour.

So the goal is not maximum transparency. The goal is the right kind of visibility: enough to prevent catastrophe, not so much that it triggers pre-emption and war.

Radical Interpretability in simple terms

Radical interpretability turns minds into auditable objects – that can stabilise cooperation and sharpen ethics.
It can also trigger an arms race between mind-reading and mind-shielding.
A sane long-term equilibrium may use selective transparency: prove safety and treaty compliance without exposing everything.

What motivated me in this like of thinking is wanting a future where cooperation beats defection, and I’m not saying that radical interpretability is a silver bullet – though I do believe it would be a powerful tool that must be paired with the right norms (and institutions regulating them) that make transparency safer than secrecy.

Bluntly: If we can read minds, we can build peace. If we can’t govern mind-reading, we can also build hell.

What would a radically interpretable future look like?

Suppose radical interpretability is eventually solved – not by twenty-first-century neuroscientists, but by civilisations with vastly deeper science and ultraintelligent researchers. What sort of world follows?

One possibility is that language ceases to be the main interface between minds. Language is a narrow channel – I compress part of my thought into a sequence of words, you hear those words, and your brain attempts to reconstruct something resembling the thought I began with. Mature minds might instead exchange structured cognitive states directly, with translation systems mapping between radically different architectures. Spoken language could survive for the same reason handwriting survives email, but it might no longer be the highest-bandwidth way intelligent agents understand one another.

Another possibility is that civilisations deliberately build auditable minds. An artificial mind need not expose everything it thinks. It could provide trusted interfaces through which another agent can inspect selected beliefs, commitments or safety-relevant processes. Instead of saying “trust me”, it could provide evidence about the machinery generating its behaviour.

At the same time, privacy would become an engineering problem rather than a fact of nature. Minds might compartmentalise private thought, encrypt sensitive computation or reveal proofs about themselves without revealing the underlying information. Radical interpretability and radical privacy technology could advance together.

The strategic effects could be profound. Combine readable minds with the transparency of history – the possibility that advanced civilisations can reconstruct much of one another’s past from enduring physical traces – and deception becomes harder in two directions. A civilisation may struggle both to conceal what it currently intends and to rewrite what it previously did.

This is the setting I have elsewhere called cosmic auditability. A civilisation encountering another may be judged not merely by its diplomatic message but by its architecture, history, treatment of weaker beings, previous commitments and evidence about the objectives it actually pursued.

None of this guarantees a peaceful universe. Readability can aid coercion as easily as cooperation, and deliberately opaque agents may remain possible. But it changes the strategic landscape. Verified commitments become easier; persistent deception becomes harder; reputational information becomes richer; and coalitions of cooperators have better evidence on which to decide whom to trust.

At the furthest edge of the idea, radical interpretability could also reshape ethics itself. If mature minds can inspect not only beliefs and plans but the physical basis of suffering and flourishing, disagreements about welfare could become substantially more empirical. Moral debate would not disappear, but some of the facts over which we currently grope in the dark might become directly measurable.

In that world, interpretability would no longer be a specialist branch of AI safety. It would be part of the infrastructure of civilisation.

Finale

Radical interpretability would not abolish conflict, deception or moral disagreement. It would change their economics.

Hidden intentions could become harder to sustain. Verifiable commitments could become cheaper. Privacy could become something minds deliberately engineer rather than something they receive automatically. Facts about suffering and wellbeing might become far more accessible than they are today.

The important goal is therefore auditability, not omniscience. A mature civilisation need not know every passing thought in another mind. It may only need enough access – or sufficiently trustworthy proofs – to distinguish cooperation from predation before that distinction is settled by war.

If that becomes possible, mind-reading may turn out to matter less as a surveillance technology than as one of the infrastructures of mature civilisation.

Footnotes

1. radical interpretability may sound like science fiction, but notice the existence of ordinary (non mystical) steps involved in modern AI interpretability. In broad strokes, it’s like diagnosing a mechanical issue in a car – you can open the hood, observe what’s going on underneath, and diagnose issues. So with minds (like AI), instead of guessing what is being thought from its output i.e. words, one can look inside at the moving parts and derive a detailed map of: what it believes, what it wants, what it is planning, which facts would change its mind etc. ↩︎
2. It may become the standard way of communicating in the long term future. ↩︎
3. My mother told me that when I was learning to talk, I once pointed at a cow and said “dog!”. ↩︎
4. Tang et al. reconstructed continuous semantic content using non-invasive fMRI, and importantly found that successful decoding depended on subject cooperation. And Fan L. Cheng et al. has reconstructed visual percepts from human brain activity. Also engineers can sometimes locate a circuit in a model that tracks a feature like “toxicity” or “gender” or “a cat’s face”. ↩︎
5. See our interview with Jesse Hoogland on singular learning theory and developmental interpretability. Also the paper Deep Learning is Singular, and That’s Good by Daniel Murfet et al. ↩︎
6. aka pre-emptive retaliation – think of all the horrors that have been enacted in the name of ‘striking first, just in case‘. ↩︎
7. See the Orthogonality thesis post at Less Wrong, and more recently the post Orthogonality is not a Forecast. ↩︎
8. See the Landscape of Value concept in the ‘Value Space‘ and the ‘Understanding V-Risk: Navigating the Complex Landscape of Value in AI‘ posts. ↩︎
9. See Anders Sandberg’s writing – Ethics of brain emulations. ↩︎
10. See discussions on bubble worlds for more… ↩︎
11. See Sharon Hewitt Rawlette’s writing on The Feeling of Value. ↩︎
12. See the interview on ‘Valence Realism, Consciousness & AI – Andrés Gómez Emilsson‘ ↩︎