Objective Moral Ontology Should Inform AI Alignment

I lean moral naturalist: the view that morality is not a fictional or supernatural layer placed over the world, but is grounded in real features of reality. Morality and purpose, on this view, emerge from a rationally sound understanding of what the universe contains: conscious systems, valenced experience, welfare, agency, flourishing, suffering, and interests.

The mistake I think many humans often make, including rationalists, sceptics, and humanists, is to locate the root of moral ontology inside human preference, agreement, culture, or volition. Even after absorbing both Sigmund Freud’s and Carl Sagan’s “great demotions” – the Copernican, Darwinian, Freudian, and related blows to human centrality – we can still quietly, often unknowingly, smuggle in humanity to take sole position at the centre of value.¹

This seems wrong.

Moral value is not created by human endorsement. It is grounded in real features of reality, especially the valenced states and interests of conscious systems. Human judgement is one imperfect epistemic route to discovering, interpreting, and coordinating around those values. It is not the source of those values.

I want to be clear that this is not a mystical claim. There are physical systems that instantiate conscious states. Some of those states are better or worse for the subject. Moral facts, if they exist in this sense, supervene on those welfare-relevant facts.

That does not require floating moral ectoplasm.²

It only requires that:
– sentience is real;
– valence is real;
– some states are bad-for or good-for the subject;
– impartial reason has to take those facts seriously.

My central claim is that experience contains intrinsic normative significance. Pain is not bad because humans vote it bad, neither is suffering bad because some society disapproves of it. A terrified animal, a tormented octopus, or a future digital mind would not need human endorsement for its suffering to matter. The suffering matters because of what it is like from the inside, or because of the kind of damaged, frustrated, negatively valenced state it constitutes.³

This matters for AI alignment. If value is rooted only in human volition, then AI alignment collapses into preference management, bargaining, power aggregation, or “what do humans currently want?” That is too thin a foundation. Humans often want terrible things. We can want cruel, confused, parochial, status-driven, and self-defeating things. A morally serious AI should treat human preferences as morally relevant evidence, not as sovereign commands.

Perhaps it is not enough to stop at saying “the root node of the hierarchy of ontology should sit in reality” – it helps if we specify which features of reality are value-bearing and how they generate reasons else it risks sounding grand but underspecified. The phrase “real features in the universe” does useful work after we specify what kind of features: valenced consciousness, suffering, pleasure, agency, preference frustration, flourishing, capability, social relation, meaning, autonomy, and perhaps other goods we have not yet clearly identified.⁴

It helps to separate three questions that are often run together.

Moral epistemology asks how we know, or justifiably believe, moral truths.

Moral semantics asks what moral words like “good”, “wrong”, “harm”, “flourishing”, and “ought” mean.

Distinguishing moral ontology, moral epistemology, and moral semantics:
- Moral ontology asks: what makes moral claims true, if anything?
- Moral epistemology asks: how do we know or justifiably believe moral truths?
- Moral semantics asks: what do moral words like “good”, “wrong”, “harm”, “flourishing”, and “ought” mean?

My claim is mainly ontological: value is not made by human endorsement; it is grounded in welfare-relevant features of conscious life. But to justify this claim, we need epistemology. We need to know how to identify those features, compare them, and avoid fooling ourselves. That is where reason, evidence, science, phenomenology, neuroscience, decision theory, and reflective equilibrium come in.

Ok, so far so good? Let’s reflect on all this – several complications need to be faced rather than politely hidden under the rug.

First, the is/ought gap does not disappear just because suffering is real. Pain exists. Suffering exists. But why does its existence generate an “ought”? We need a bridge principle: something like “states of intense negative valence are pro tanto bad”, or “there is reason to reduce suffering where possible, all else equal”.

To some people, that principle may seem self-evident. Others may treat it as non-natural, naturalistically grounded, constructivist, or conceptually built into what we mean by welfare. But we cannot simply point at suffering and declare that normativity automatically falls out. I wish it did. In some sense, it almost does. But philosophically, “almost” is where the bodies are buried.⁵

Second, “experience has intrinsic value” can slide too quickly into a narrow form hedonism.⁶ Pure pleasure and suffering matter enormously, perhaps centrally. But are they the whole story?

Suppose a person is blissfully deceived, stripped of agency, disconnected from reality, and kept in an experience machine. Is that as good as a life of real understanding, relationship, achievement, and autonomy? If not, value is not exhausted by valence. We need room for agency, truth-contact, integrity, meaning, freedom from manipulation, and perhaps objective goods.

This also matters for advanced AI. Some discussions of superintelligence imagine a utilitronium shockwave: a future in which matter is converted into vast quantities of narrow bliss, “quivering bliss-jelly” spreading across the light cone.⁷ I take the concern seriously, but I do not think it settles the matter. If welfare includes agency, knowledge, autonomy, relationship, and integrity, then maximising raw positive valence may be a grotesque simplification rather than the logical endpoint of ethics.

Third, different minds may not sit neatly on one human-like pleasure-pain scale. Octopus suffering likely differs from human suffering.⁸ Insect valence, if present, may be stranger still. Future AI phenomenology, if it exists, could be very unlike ours.

So the ontology should not be “human-like suffering matters wherever copied”. It should be more abstract: morally relevant systems may instantiate valenced or interest-bearing states in different architectures. The hard problem⁹ is mapping those states without either anthropomorphising them or dismissing them.

So yes: we should not assume octopus suffering feels like human suffering, and blithely infer that if it does not, it matters less or not at all. We should replace the inference “therefore it may not matter” with something like “different physical implementation, uncertain phenomenology, still plausible welfare significance” (given different physical implementation from ours and uncertain phenomenology there is still plausible welfare significance).¹⁰

Fourth, not every pattern has moral status. Everything is a pattern if you squint hard enough: blog posts, weather systems, economies, ecosystems, language models, and late-night pub conversations about AI philosophy that should probably have ended an hour earlier. The relevant patterns must be the right kind: integrated, self-maintaining, information-processing, valence-bearing, agency-related, and welfare-relevant. Otherwise we risk inflating moral status everywhere.¹¹

Fifth, we should be cautious about saying that intrinsic value exists in “life” as such. Life as a broad category may not be the true value-bearer. Conscious life, sentient life, or welfare-capable life is the better target.

Bacteria are alive, but probably not subjects of experience. Plants may have functional interests in a biological sense, but it is not obvious that they have welfare in the morally weighty sense. Artificial sentience, if it emerges, may or may not be “life” in the biological sense, but that should not decide whether it matters morally. If we use “life” too broadly, we invite avoidable confusion.

Sixth, impartiality does not mean flattening all distinctions. A morality grounded in sentient welfare may still give special weight to intensity, duration, cognitive sophistication, agency, relationship, consent, vulnerability, future potential, and rights-like constraints. “All morally relevant beings matter” does not mean “all claims have equal weight”. That distinction matters. Without it, the view becomes unusable.

This is why moral realism does not make AI alignment easy. It makes the target less arbitrary, but not simple. Even if moral ontology is objectively grounded, alignment remains hard because moral epistemology, moral aggregation, institutional design, uncertainty, and conflict remain hard. But “hard” is not the same as “intractable”.¹²

Cultural relativism also looks weaker from this perspective. Societies can be wrong – if a culture normalises cruelty, domination, or animal suffering, that does not make those practices morally fine. Human consensus is not a magic laundering machine that turns bad things into permissible things.

The same applies to preference aggregation. Preferences are not automatically authoritative. A sadist’s pleasure in another’s pain counts as a psychological fact, but not as a clean moral reason to produce suffering. Some preferences are evidence of welfare. Others are corruptions, addictions, status games, trauma patterns, or evolved kludges.

Moral ontology having it’s feet firmly rooted in objective grounding gives AI ethics a better target than “do what users want”. A powerful AI should not simply maximise revealed preferences, market demand, political commands, or elite instructions. It should model what actually matters: suffering, flourishing, agency, justice, epistemic integrity, ecological and animal welfare, future generations, and the conditions under which beings can develop well.

It also explains why moral progress is possible, rather than merely moral drift. If values are only created by societies, then “progress” becomes hard to distinguish from “change we currently like”. But if there are stance-independent welfare facts, then abolition, women’s rights, disability rights, animal welfare, and concern for future generations can be understood as partial discoveries or corrections. They are not just fashionable rearrangements of sentiment.

Conclusion

In the objective sense, intrinsic values are not located in human minds as acts of endorsement. They are grounded in welfare-relevant features of conscious and agency-capable systems. Human minds participate in real value by experiencing, recognising, reasoning about, and responding to those features. Moral inquiry is the attempt to bring our concepts, institutions, and motivations into better contact with those facts.

That gives us an ontology while leaving room for epistemic humility.

For AI, the implication is important: democratic input, human preference learning, constitutional AI, deliberation, and governance are not the source of morality. They are fallible instruments for tracking, approximating, and operationalising morality under uncertainty – they matter, but they do not get the final metaphysical word.

A serious theory of moral reasons needs at least two things: value-bearing features and an account of how those features generate reasons for action.

The relevant features may include suffering, bliss, wellbeing, agency, autonomy, understanding, meaning, relationship, and perhaps novelty.¹³

The reasons explain why and how those features should guide action.

A being’s suffering gives any sufficiently capable agent a pro tanto reason to reduce it, because suffering is a real negative condition for the subject undergoing it. That reason can be outweighed by other reasons, but it is not created by anyone’s approval.

A being’s suffering gives any sufficiently capable agent a pro tanto reason to reduce it, because suffering is a real negative condition for the subject undergoing it. That reason can be outweighed by other reasons, but it is not created by anyone’s approval.

This is the hinge. People will disagree about exactly how it swings. I do not pretend that all of the hard problems can be solved in a blog post. But the position is serious, defensible, and better than the alternative I am criticising: treating human preference as the root of value, rather than as one morally messy data stream among others.

So is morality spooky, out there somewhere haunting the fabric of the universe?

No.

Some configurations of reality are better or worse for the subjects embedded in them. Consciousness, valence, agency, and flourishing are not projections from human culture. They are real phenomena. Morality is the rational, impartial, action-guiding response to those phenomena.

Footnotes

1. Some modern philosophers argue that AI is now delivering a fourth narcissistic wound to humanity by challenging our unique claim to intelligence and creativity. Philosophers and techno-sociologists – notably Luciano Floridi – argue that AI represents a profound demotion of human uniqueness by shattering our monopoly on information processing, reasoning, and creativity. I argue that we may soon face a further demotion, this time in the moral domain. ↩︎
2. Moral ectoplasm? No. Moral value is not a separate magical ingredient; it is a higher-level property of real systems, like health, information, computation, fragility, or harm. “Health” is not spooky, even though it is not reducible to one molecule. It depends on how many biological systems are functioning relative to the organism’s needs. Likewise, “suffering is bad” may depend on how conscious systems instantiate aversive experience relative to their welfare. ↩︎
3. This is a strong anti-anthropocentric stance – see my posts More Moral Than Us, and The Knowledge Argument Applied to Ethics etc ↩︎
4. I’m not claiming to be right about the features, or to know them all. I argue for leveraging powerful AI to help discover what’s valuable, and what norms to derive from what’s valuable though indirect normativity. ↩︎
5. The point is not that the is/ought gap refutes moral realism. It is that any serious moral realist needs an account of how natural facts generate normative reasons, rather than merely asserting that they do. ↩︎
6. Not to suggest hedonism is actually narrow – some think that experiential quality includes novelty, texture etc – read Nick Bostrom’s Deep Utopia – just do it, trust me it’s worth it. ↩︎
7. Negative utilitarian philosopher David Pearce has warned that classical utilitarianism may imply a utilitronium shockwave. I am not as confident in this conclusion as he is, and we have discussed our differences online. Note: I have David to thank for a lot of my motivation for what I do now. ↩︎
8. Cephalopod nervous systems evolved independently from vertebrate nervous systems, with complex convergent features but very different anatomical organisation. Reviews describe cephalopod brains as large, complex, and partly analogous to vertebrate structures, but not straightforwardly homologous to them. Their arms also contain a massive distributed nervous system, with more neurons distributed across the arms than in the brain, and recent work shows modular or segmented organisation in the axial nerve cords. ↩︎
9. This is not David Chalmers’ “hard problem of consciousness”, which concerns why and how physical processes give rise to subjective experience at all. The problem here is different: given that some systems have morally relevant states, how do we identify, compare, and map those states across very different kinds of minds without anthropomorphising them or dismissing them? ↩︎
10. It is not the human pattern of pain that is morally fundamental. What matters is that negatively valenced states occur in subjects, realised through that subject’s own architecture. Human pain, octopus pain, crustacean distress, and possible digital suffering may be physically different while still sharing the higher-level moral property of being bad-for the subject. ↩︎
11. This does not require settling every dispute with panpsychism. The narrower point is that morally weighty systems need more than mere structure or complexity. If consciousness, valence, and welfare are doing the moral work, then we need criteria for identifying where they are plausibly present. ↩︎
12. This connects with my argument in “AI Alignment to Moral Realism”: that AI alignment should not stop at modelling human preference, but should aim at tracking what is genuinely valuable under deep uncertainty. ↩︎
13. Novelty is not obviously intrinsically valuable. It may be valuable when it contributes to discovery, growth, play, meaning, beauty, or flourishing. But novelty by itself includes pointless chaos, instability, and fresh new forms of misery (nature is already quite inventive in that department). Novelty is a candidate value, not a foundation, unless it is integrated into a broader account of flourishing. ↩︎