Morality by mathematical precision, or software design?

Some argue that without mathematical precision AI safety will come down to a cosmic roll of the dice with steep odds of failing – I hope they are wrong.

The debate over whether AI morality should be achieved through mathematical precision or software design reveals a tension over whether human values can be perfectly specified and formalised.

The argument for mathematical precision (formal specification)

Proponents of mathematical precision argue that the immense power of a superintelligence demands an equally precise moral framework. If the AI’s core goal function contains even a tiny ambiguity, a maximising agent will exploit that flaw to disastrous, unintended ends.

The Risk of Value Erosion: If the AI’s goal is defined through fuzzy heuristics or observational learning, the argument is that the true “value” we want to preserve (i.e. human flourishing, consciousness, etc.) is too fragile. A slight miscalculation could result in the AI optimising for a proxy (like paperclips, or maximising one simple sensation) instead of the rich complexity of human life.

Safety as Formal Verification: This approach sees AI safety more generally as a problem of formal verification. The morality must be encoded as a clear, non-negotiable utility function or set of axioms that a future ASI can interpret the real meaning of perfectly. If we cannot formally prove the safety of the initial seed values, the entire project is deemed too risky – a cosmic roll of the dice.

The argument for software design (i.e. via indirect normativity and value learning)

Opponents of the purely mathematical approach argue that human morality is too complex (high Kolmogorov complexity) and context-dependent to ever be fully compressed into a concise set of equations. Instead, morality must be an emergent quality of robust software design.

Human Values are Emergent: Human values were not derived from a single equation; they emerged from billions of years of evolution, social interaction, and history. This camp favours Indirect Normativity – designing an AI that is motivated to discover the correct ethical framework, rather than being handed one.

Mechanism vs. Content: The focus shifts from specifying the content of morality (e.g., “Utilitarianism is the answer”) to designing the mechanism for ethical reasoning (e.g., “Build an AI that can perfectly apply moral imagination, impartiality, and rational reflection to any problem”).

The Interpretive Bridge: This approach acknowledges that the AI’s eventual ethical reasoning might be alien, but emphasises the need for an interpretive bridge – a translation layer that allows the AI to explain its hyper-rational moral choices in a way that is intelligible and trustworthy to human users, allowing for continuous human oversight and correction (a “moral loop”).

Ultimately, the choice is between attempting to lock in a single, unshakeable definition of good at the outset, or creating a highly robust, self-improving system that can learn, evolve, and converge on the best moral solution over time.

Which seems most plausible to you?