Terence Cuneo on Epistemic and Moral Realism and Whether a Machine Could Have Moral Knowledge
Terence Cuneo is Marsh Professor of Intellectual and Moral Philosophy at the University of Vermont. He is best known for The Normative Web (Oxford, 2007), which argues for moral realism by an indirect route, and more recently for The Moral Universe (Oxford, 2024), written with John Bengson and Russ Shafer-Landau. The three are now at work on a further book, Grasping Morality, on how anyone comes to know moral facts in the first place. He told me it is at least four years away.
I invited Terence for interview because the question I keep returning to is a question about moral knowledge, and he has spent his career on it seemingly without ever being asked to apply it outside the human case. As far as I can tell, this interview is the first recording where he has addressed the topic of AI.
I later discovered that Terence is also an amazing guitarist!
The ‘companions in guilt’ argument Terence Cuneo is well known for
Start with the complaint that moral facts are strange. J. L. Mackie put it most memorably: if there really were facts about right and wrong, out there, independent of what anyone thinks, they would be very odd items indeed.1 They would not obviously cause anything. It is hard to say how we would ever get hold of them. And it is unclear why grasping one should move anybody to do anything.
Cuneo’s response is that every one of those complaints applies just as well to the epistemic domain, meaning facts about what anyone has good reason to believe. If moral facts are too strange to exist, then so are facts about whether a belief is justified, or whether a line of inquiry is worth pursuing. And denying those, he argues, is not something anyone can actually live with. He described it in the interview as a kind of Armageddon for rational agency2, since the whole business of inquiry, science included, runs on there being reasons to believe some things rather than others.
One thing he seemed firm about is that he does not think of this as a defensive manoeuvre – in that it is meant to be a positive argument that moral (and epistemic) facts exist, rather than a way of embarrassing the sceptic into silence.
What is morality?
The Moral Universe goes further than the earlier book and says what morality is, on a view called non-naturalism, which holds that moral properties are not reducible to the sort of properties that physics or biology traffic in. The distinctive move is an appeal to essences – which is not merely a necessary truth that killing people for entertainment is wrong, it goes beyond that – it belongs to the essence of that kind of act that it is wrong, in something like the way that theologians have said it belongs to the essence of God to be omniscient rather than merely being a brute fact that God happens to be. Unfortunately we didn’t have time to discuss the Euthyphro dilemma and how it relates to AI alignment.
Alongside this sit what he and Shafer-Landau call the moral fixed points that hold of necessity and that anyone with a reasonable grasp of the relevant concepts is in a position to see. Small children can possess the concept, but a reasonable grasp of fixed points is to have some mastery of them, which takes training and practice.
As for how any of this gets installed in our minds, the answer is intuition, though not in the ‘gut feelings’ or ‘snap judgements’ sense – Terence means conscious, non-sensory states in which something is presented to you as being the case. That a particular act of cruelty is atrocious can strike you that way. Whether such states are trustworthy is a further question, and their answer runs through an account of what makes a cognitive practice reliable.
On Artificial Intelligence
At the time of recording, Cuneo has written nothing about artificial intelligence and said so several times, more or less cheerfully, warning me in advance that his answers would be disappointing – but they were not.
On whether a system could grasp a moral truth rather than simply report that humans assert it, he saw no reason in principle why not, at least for the truths that follow from the concepts. If some moral claims hold in virtue of what the concepts involve, then something with a reasonable mastery of those concepts is in a position to work them out. Whether current systems have anything worth calling mastery, given the inconsistent material they are trained on, he did not claim to know.
The less encouraging side is on moral intuitions. Intuitions as he understands them are conscious states with a phenomenal character, and if these systems have no phenomenal states then they have no intuitions – if AI can’t be conscious, then this closes off the route he takes to be the principal one.3
I put to him a transposed debunking argument: our moral intuitions came from evolution, evolution was not aiming at moral truth, so perhaps our human intuitions are unreliable, and an AI trained on human moral judgement inherits whatever is wrong with our intuitions.4 The evolutionary argument depends on an empirical claim that our judgement-forming capacities are saturated by selection pressure with nothing pushing back, and he thinks that claim is unsupported and probably unsupportable by the sciences, which do not deal in moral facts at all. If the material fed into LLMs via RLHF comes from human moral thought that has already been worked over by principled reflection, then the systems are in reasonable shape, and might even reach conclusions we have not.
Then the question I have been putting to a series of philosophers, including David Enoch: could AI be more moral than we are? His answer was no to current LLMs, and the reason he gave is that being morally better takes two things, having accurate and nuanced views (which he is willing to entertain), and having conduct that reliably expresses them – which he rules out on the grounds that these systems currently have neither the agency nor anything he could recognise as a motivational state. He was careful to add that this implies nothing flattering about how good humans are at morality, only that the comparison cannot be made.5
I find this quite validating because I also argue that artificial systems may come to reason about morality better than we do while lacking any motivation to care about what they conclude. Cuneo has never written about AI, was not led towards that distinction, and drew it anyway, from his own metaethical position.
A gap between experience and normativity?
Early on I put forward the idea that agony could be inherently aversive and pleasure inherently attractive, and that this might be enough to locate value in experience without any further apparatus – he said these descriptions are purely psychological and that they tell you what states creatures are in and what they are disposed to do, and they contain nothing normative. He argues you cannot get from the fact that a creature recoils from pain to the conclusion that the pain is bad, or that there is reason not to inflict it, without adding something, and what you add is the part in dispute.
I am not persuaded either way, but I think he identified a gap at the point where description is supposed to yield normativity, and if it is a gap it requires closing rather than stepping over. But how to close that gap?6
That exchange is around the twenty-nine minute mark and is worth watching even if you skip everything else.
His closing suggestion
Asked what someone building these systems should take from his work, he offered a great idea. The current situation in metaethics, as he sees it, is that his own position has been worked out at length while its rivals have not. Expressivism, error theory and moral naturalism have all had interesting work done on them, but nothing comprehensive to the same standard. So one thing AI could usefully do is build out the rival views properly, to the point where the theories can finally be compared against each other rather than each being defended by its partisans. Whether that would change anything about morality on the ground he was unwilling to guess.
That exchange inspired me to revisit Sharon Hewitt Rawlette’s arguments in her paper Normative Qualia and a Robust Moral Realism and book The Feeling of Value, the most sustained answer I have found to the objection Terence raised. I hope to interview her in the future, perhaps next year.
Video chapters
0:00 Introduction
1:38 Normative domains: moral, epistemic, aesthetic, prudential
6:59 Does the argument prove realism, or only embarrass the sceptic?
9:03 Are moral facts strange? Mackie’s queerness objection
16:15 Motivation: do moral facts move us in a way epistemic facts don’t?
18:07 Epistemic nihilism, or Armageddon for rational agency
21:12 Revisiting the argument after twenty years
24:01 Morality flowing from the natures of things
29:30 Is agony intrinsically bad? Descriptive versus normative
34:32 Are moral facts causally inert?
42:00 Moral facts as configuring causes
43:29 First AI question: would moral facts motivate a machine?
45:43 Can we trust moral intuitions?
48:14 What an intuition actually is
52:58 The trustworthiness criterion
58:17 The moral fixed points
1:05:22 Mastery of concepts and grasping moral truth
1:07:36 Do moral reasoners have to be human?
1:11:34 Grasping versus reporting: could an AI work it out?
1:19:09 A moral Olympiad, and how you would grade it
1:21:47 Indirect normativity and Bostrom
1:24:48 Evolutionary debunking, and whether it transfers to training
1:30:38 Could AI be more moral than humans?
1:35:39 What should AI alignment take from moral realism?
1:41:08 Pluralism, and what metaethics could offer
Papers: Trusting Moral Intuitions and The Moral Fixed Points.
Books: The Normative Web and The Moral Universe + analysis symposium.
Footnotes
- J.L. Mackie argued that objective moral facts are strange, or “queer,” because they do not fit into our normal scientific view of the natural world. This idea is known as his argument from queerness, which supports his moral error theory. ↩︎
- I uploaded a short video snippet on epistemic nihilism – Armageddon for rational agency idea. ↩︎
- If an AI lacks subjective experience, its “intuitions” aren’t grounding to anything that gives moral authority. Though some philosophers question whether intuitions are inherently phenomenal, and some question whether phenomenal intuition is necessary for moral knowledge.
I don’t know whether non-conscious moral intuition is enough for alignment – but I’ve argued that ‘zombie’ AI may, for instrumental reasons, engineer itself to become sentient, or to acquire the architecture which could afford whatever it is that would be required if non-conscious moral intuition is not enough. ↩︎ - Our moral judgements may be epistemically unreliable. If there are epistemic and moral urgencies outside our current human moral Overton window, RLHF may be epistemically and morally conservative even when it seems behaviourally successful to us. As such, in that RLHF can reward an AI for remaining inside the moral Overton window when superior reasoning would sometimes require leaving it. This highlights the case for epistemic and moral uncertainty, indirect normativity, corrigibility and resistance to lock-in.
– What is more likely to produce a more morally competent agent?
– Which training genealogy gives an agent the strongest reason to expect its moral beliefs to track whatever actually makes moral claims correct?
– And, importantly, how do we transfer enough of our moral knowledge to bootstrap moral inquiry without freezing our present errors into the successor system?
On the other hand, if Sharon Street is right that evolutionary influence makes realist moral knowledge untenable, then what is RLHF approximating, and what warrants one extrapolation procedure over another? ↩︎ - David Enoch answers the question ‘Could AI become more moral than us?‘. Also see: Eric Sampson discussing whether AI should align with objective ethics, A. C. Grayling on AI and Moral Judgement, and Wendell Wallach on whether AI can be moral. ↩︎
- There are four strategies to close the gap between experience and normativity on offer, and they are not equally useful.
1. The most developed is Sharon Hewitt Rawlette’s, in The Feeling of Value: Moral Realism Grounded in Phenomenal Consciousness (2016). Her claim is that pleasure and pain are not evidence of value and disvalue but instantiations of them, and that our normative concepts get their content from acquaintance with those qualities in the first place. That blocks Terence’s objection by denying its premise rather than answering it: the description was never merely psychological, because the awfulness is a feature of how the state presents itself, and the recoiling is downstream of that rather than what the badness consists in. What makes this reply interesting is that it borrows his own epistemology. He holds that intuitions are conscious states in which something is presented to you as being the case. If agony is such a state, it presents its own badness, and he then owes an account of why one presentational state can deliver normative content and the other cannot.
2. The mainstream naturalist answer is a different one. Cornell realism, associated with Richard Boyd, David Brink and Nicholas Sturgeon, along with Peter Railton’s version, does not try to analyse goodness into natural terms at all. It holds that moral terms pick out natural properties the way “water” picks out H2O, as a discovery rather than a definition, so the open question argument has no traction: two concepts can differ while the properties they pick out are identical. Its standing difficulty is the Moral Twin Earth argument of Horgan and Timmons.
3. A third route, constitutivism (Korsgaard, Velleman, Katsafanas), grounds normativity in what is constitutive of agency rather than in experience. It will not help me here, since it would make an AI’s normative standing depend on whether it counts as an agent, which is the question I am trying to keep separate.
4. The fourth is a debating point rather than a position, but worth stating. Terence does not close the gap either. He holds that wrongness belongs to the essence of certain acts, which is a posit rather than a derivation. So both sides introduce a bridge that neither derives, and the real argument is about which bridge pays for itself. Mine buys epistemic access and motivational traction at the cost of confining value to experience. His buys scope and objectivity at the cost of making both moral knowledge and moral motivation harder to explain.
I lean towards Sharon’s view – but I think the honest position is that there is no consensus – and metaethics has at least one unexplained commitment. ↩︎
