Questioning Assumptions About Human Reasoning in Debates About AI Reasoning

“All that neural net does is just matrix multiplications and gradient descent”… “When we look inside, we see only statistical pattern matching, not the structured, systematic, rule-following processes we associate with reasoning.”
As far as I can tell we can’t find evidence of symbolic reasoning at the level of human neurons either.

The debate about whether AIs can reason often relies on a folk theory about human cognition that doesn’t hold up under neuroscientific scrutiny. Some say frontier AI models can’t genuinely understand or reason because we don’t know if AI is doing truly what’s operationally required for reasoning to occur (i.e. emergent compositionality¹) or if it’s doing sophisticated interpolation that feels compositional from the inside. Skeptics often say that neural nets lack the explicit symbolic structures required for understanding, and therefore don’t have the required robust compositionality to handle arbitrary novel combinations beyond training data. However, I don’t think we can yet point to the human neural net and discover this either (no symbol module or rule processor sitting in our neurons), yet we feel like humans do symbolic manipulation at higher levels – in specialised structures dealing with language, working memory, kinesthetic awareness etc.

I wonder if all that humans are doing is emergent systematic compositionality too? And by that measure aren’t frontier AI models doing increasingly well?

The difference may be in degree/robustness rather than kind. Perhaps AI and humans are doing emergent systematic compositionality differently (different levels, degrees of robustness, degrees of symbolic grounding in multisensory perception of the outside-the-mind world) in ways that matter to the question of what counts as reasoning. Frontier models increasingly pass tests we thought required real understanding – and then skeptics often shift the goalposts (or more usefully refine criteria and come up with new tests). Meanwhile, humans also fail on variations, confabulate², and exhibit biases that look like mere pattern matching.

Since we seem to be so hard on AI systems compared to humans, a couple of important questions are:

• What behavioural failures disqualify a system (human or artificial) from reasoning and why?
• What specific forms of emergent systematic compositionality would justify calling what it is doing ‘reasoning’ & ‘understanding’?

So on one hand we have “just matrix multiplications and gradient descent”, and the other “just electrochemical signals and neurotransmitter release”. Is it fair to say there is no explicit world model or causal structure at all in either AI or human brains? No, since it looks like we are observing the wrong level. You can’t understand reasoning by only looking at the lowest implementation level. Water’s wetness isn’t visible in H2O molecules either.

Many critiques of AI reasoning capability would, if applied consistently, force us to conclude humans can’t reason either. They rely on:

1. Demanding properties we can’t find in human brains
2. Confusing implementation details with functional capabilities
3. Setting standards that human cognition itself doesn’t meet

We should question our assumptions about how the human brain is doing reasoning before concluding that AI can’t reason.

Footnotes

1. One way to think about how systems operationally scaffold reasoning capability is through the lens of “emergent compositionality” – though there doesn’t seem to be clear consensus among AI researchers that reasoning in frontier models is best described as “emergent compositionality”. While this framing seems contested, it serves to highlight some way to describe what both humans and AI might be doing that’s more sophisticated than “just neurons” but doesn’t require the “explicit symbolic structures” skeptics demand. ↩︎
2. Humans confabulate reasons for decisions made unconsciously all the time (split-brain experiments, implicit bias, etc.) ↩︎