AI & Morality – Empathy vs Compassion
Allan argues that morality can be understood as a natural, human-constructed system for regulating behaviour, not as something requiring supernatural or transcendental grounding; he also says moral systems are grounded in facts about cooperation, safety, welfare and mutual benefit, not mere personal whim or majority vote.1
If moral rules are like traffic laws, human-created systems for coordinating behaviour around shared vulnerabilities and mutual benefit, does that mean a sufficiently capable AI could become a better ‘moral traffic engineer’ than humans, by identifying harms, trade-offs, exceptions and institutional failures more clearly than we do?
Should moral systems be based on empathy?
It was raised during the discussion time that moral systems are based on empathy – though they may have meant something closer to compassion – to which I argued that the kind of moral system that I favour would be better placed if it were indeed based on compassion.
Empathy and compassion are not the same thing. Empathy is usually the capacity to feel with, simulate, or understand another’s perspective. Compassion is more motivational: concern for suffering plus some disposition to alleviate it. Empathy can be parochial, biased, exhausting, manipulable, and often strongest for nearby, vivid, relatable cases. Compassion can be broader and more stable, though it too can be partial unless disciplined by impartial principles.2
Should we really treat empathy as foundational, given that it is often partial and biased?
I’d would be careful about saying that moral systems are based on empathy, at the very least it seems under-specified. Empathy may be one psychological route into morality, but it is a poor foundation by itself. It favours the crying child in front of us over the millions unseen; the charismatic victim over the statistically represented victim; the in-group over the out-group. As Paul Bloom and others argue, empathy can be a morally distorting spotlight.3 Compassion plus impartiality gets closer to what he probably wants.
Empathy helps us notice that others matter. Compassion motivates concern for their welfare. Impartiality disciplines that concern so it is not captured by proximity, tribe, charisma, species membership, or personal preference.
How does this matter to my ‘AI may be more moral than humans‘ thesis?
An AI may not have mammalian empathy or might not “feel with” us – but it could, in principle, understand suffering, preference frustration, welfare loss, coercion, injustice, risk and trade-offs with far less parochial distortion than humans. The hard question is whether that would amount to morality, or only moral cognition without actual moral concern.
On bringing partial people into dialogue about impartial ethics
What should the function of bringing partial people into dialogue with regard to impartial ethics?
Where ethical dialogue includes partial agents, is the point of that dialogue to discover moral facts, to persuade people toward impartiality, to secure legitimacy, and/or to coordinate social rules?
Perhaps the most outstanding functional use case is that the dialogue could have an epistemic function: we expose each other to neglected facts, counterexamples, affected perspectives, hidden harms and inconsistencies. Dialogue helps partial agents become less partial by exposing them to encounter evidence and arguments they might otherwise ignore.
Also, it could have a motivational function: people are more likely to accept some moral demanding norms when they have participated in the reasoning rather than having conclusions imposed on them. This is not just soft democratic niceness; it is a practical condition for uptake.
Further, it could have a social legitimacy function: even if there are facts of the matter about harm and welfare, practically in order for rules that govern shared life to gain traction, there needs to be some public justification. Otherwise morality may feel like technocratic instruction: “The clever people, or the AI, have calculated the answer; kindly obey.” If framed this way people may reject and feel animosity toward moral norms they feel are technocratically imposed.
It could also have a coordination function: traditionally moral systems are not merely private discoveries – often they are constructed as social technologies. In order for them to work, they need common knowledge, stable expectations, shared enforcement and workable norms. So here dialogue helps build the system, not merely discover propositions.4
But here is what I think might be the main pressure point: if facts of the matter matter, then dialogue cannot just be preference aggregation. It cannot be “everyone brings their partial views and we compromise somewhere in the middle”. That would make morality too dependent on existing prejudice. If the dialogue includes racists, speciesists, nationalists, sadists, exploiters, or simply the complacent, their inclusion may be important for persuasion and legitimacy, but their partiality does not get equal epistemic weight merely because they have a viewpoint.
Semi-cognitivism, cognitive limitation and moral disagreement
Part of me thinks that non-cognitivism could be motivated by limitations of cognition – but I’m aware that the positions strong points aren’t just based on failed cognitivism where “we could have the moral facts if only we were clever enough – so let’s feel good about ourselves and just be non-cognitivists, after all, if we develop strong AI that can discover these facts, then we can later convert to cognitivism”. As such the strong version of semi-cognitivism, the aspects of it that are non-cognitivist, don’t have to rely on this framing.
A semi-cognitivist view is better understood as saying moral judgement has at least two components (both a cognitivist and a conative/expressive/practical component):
- First, a cognitive component: claims about harm, welfare, interests, consent, coercion, sentience, institutional consequences, consistency, reciprocity, social stability and so on. These are truth-apt in the ordinary sense. Allan’s own taxonomy treats cognitivism as the view that moral utterances are truth-apt, and he separately distinguishes this from realism and pluralism.
- Second, a conative / expressive / practical component: moral utterances do not merely describe; they commend, condemn, prescribe, coordinate, motivate, and recruit others into a shared normative stance. Allan explicitly has sympathy with sophisticated emotivism, while also arguing that moral objectivity involves impartial reasons rather than merely personal preference.
So the non-cognitivist residue is not charitably framed as “cognition running out of steam”. It is that morality is not just a map of facts; it is also a system for action-guidance. “This causes suffering” is descriptive. “This suffering counts against the act” is already doing normative work. “You ought not do it” is doing even more: it is guiding, demanding, coordinating, and potentially feels like blaming (which is fine if it’s justified). Note, my take is that some moral facts are inherently valuable and dis-valuable (ceteris parabis, pleasure and pain) – and hence are also inherently action guiding.5
That said, cognitive limitations do explain a lot of moral disagreement. Humans disagree because we misperceive facts, reason badly, discount distant consequences, privilege vivid victims, protect status, rationalise tribal loyalties, etc. But even if those epistemic defects were removed, there may still be disagreement about weightings: liberty versus welfare, equality versus aggregate benefit, desert versus mercy, sanctity versus autonomy, near lives versus future lives, actual beings versus possible beings.
Would there be moral convergence across ultra intelligent moral agents?
I think the moral convergence hypothesis is worth it’s salt – there could be most-to-total convergence across ultra-intelligent agents if adequate representations of moral facts and sound reasoning around them could be adequately compressed enough for cognitive competence – though this may turn on motivation – perhaps the ultra-intelligent agents would naturally become motivated by merely moral cognitive competence, or perhaps cognitive competence itself wouldn’t be enough, in that they would have to adequately motivated to care aside from being cognitively competent wrt morality.
Alternatively in a universe where full knowledge of morality was really really hard – there could be substantial but not total convergence among sufficiently rational agents, assuming they understand morality at least partly (and weren’t just brute paperclip maximisers).
They would likely converge on many thin moral constraints: arbitrary partiality is suspect; suffering and extreme preference frustration matter; coercion needs justification; like cases require like treatment; hidden interests should be made public; evidence should constrain moral judgement; and norms should be assessed by their effects on agents capable of welfare, agency or experience. This is close to Allan’s claim that moral reasons must appeal beyond the agent’s own interests or preferred group, and that objectivity in ethics is better understood as opposition to biased and prejudicial reasoning rather than access to “spooky metaphysical furniture”.6
They may also converge on many practical norms because the world constrains viable ethics. Any society of vulnerable, interacting agents needs norms around violence, deception, trust, property/use, care, reciprocity, dispute resolution, punishment, promise-keeping, and protection of the weak. Not because these norms float in Platonic space, but because social life becomes unworkable without some versions of them. Traffic laws are conventional in detail, but not arbitrary in function.
Where my convergence thesis may become weaker is at the level of deep normative theory. Rational agents might still disagree about population ethics, aggregation, rights, side-constraints, moral uncertainty, risk, identity, desert, and how to compare radically different kinds of welfare. If there is a single correct answer, it is not obvious that rationality alone gets you there quickly. Moral philosophy has had a few millennia and still looks less like physics and in some cases more like a polite duel with red-hot pokers.
AI could plausibly outperform humans on the cognitive side of morality: gathering facts, modelling consequences, detecting inconsistency, correcting parochial bias, identifying affected parties, and comparing institutional designs. That could produce convergence toward more impartial moral judgement. But (I argue) cognitive moral competency is just one aspect of being ideally moral – as such it wouldn’t be more moral, unless the AI also has the right motivational orientation toward moral patients.7 8
So yes, under a partly cognitivist, impartial, fact-sensitive account, I think there should be real convergence pressure among rational agents. But not because rationality mechanically spits out utilitarianism, Kantianism, contractualism, or Allan’s favoured formulation. Rather, rationality progressively removes bad disagreement: ignorance, bias, inconsistency, motivated reasoning, local prejudice, and arbitrary privileging.
Footnotes
- See Is Morality Natural or Supernatural? by Leslie Allan ↩︎
- See my recent interview with Magnus Vinding on his new book “Compassionate Purpose” – which I highly recommend reading. ↩︎
- See Paul Bloom’s book “Against Empathy: The Case for Rational Compassion” – argues that emotional empathy is a poor moral guide that often leads to biased, short-sighted, and unfair decisions. Instead of discarding care for others, Bloom advocates for rational compassion – a combination of a reasoned desire to do good with detached, logical evaluation. ↩︎
- Though I am very partial to the idea that powerful AI could discover moral truth through understanding natural systems, what moral features supervene on non-moral features, and the soundness of reasoning to put it all into place in a non-arbitrary way. The discovery function of the social aspects of moral discovery and progress may me simulated across many branches at break-neck speed and incredible depth – and the discovery function may not exclusively have to be achieved through social aspects. ↩︎
- See Sharon Hewitt Rawlette’s book The Feeling of Value and the 80k Hours podcast with Sharon on why pleasure and pain are the only things that intrinsically matter. ↩︎
- Though I’m not spooked by metaphysical furniture – reality is what it is, what is more spooky to me is inadequately grounded constructions of morality that don’t quite pass muster, i.e. because their justifications are circular or disappear into obscure clouds of arbitrariness rather than terminate on justified axioms or grounded empirical truths. ↩︎
- See post Capability Control vs Motivation Selection ↩︎
- Under some conceptions of morality, there may also be a need for the right authority relation to those governed by its judgements. ↩︎
