Training a Wikipedia Manual of Style Assistant with RLHF
How RLHF Could Train an AI Assistant to Follow Wikipedia’s Manual of Style
Reinforcement Learning from Human Feedback (RLHF) is a post-training pipeline that uses human judgements to shape a pre-trained language model’s behaviour. Starting from a base large language model (LLM), RLHF can be used to train the model into a helpful assistant that answers user queries while conforming to a target standard. In this case, the target standard is Wikipedia’s Manual of Style (WMoS).1
Beyond general helpfulness, the assistant should produce clear, neutral, well-structured, encyclopaedic responses that observe Wikipedia’s norms concerning tone, article structure, dates, titles, linking, quotations, punctuation, and the avoidance of promotional, tendentious, or editorially biased language. It should answer in a way that resembles the work of a competent Wikipedia editor.
Beyond WMoS, a serious Wikipedia assistant would also need to respect policies such as Neutral Point of View, Verifiability, No Original Research, and Biographies of Living Persons. Even so, the WMoS is a useful concrete target because it contains many detailed editorial norms that are hard to specify exhaustively in a conventional software system. That makes it a good example of why human feedback is useful in the first place.
The basic RLHF pipeline
The usual RLHF process has three broad stages. First, the model is trained by supervised fine-tuning on demonstrations of desired behaviour. Second, humans compare or rank model outputs, and those comparisons are used to train a reward model. Third, the language model is further optimised against that reward model using a reinforcement learning algorithm, commonly Proximal Policy Optimisation (PPO), often with a constraint that prevents the model from drifting too far from its earlier behaviour.2
For a Wikipedia-style assistant, each stage would need to be designed with the WMoS in mind. It would not be enough to train the model to sound polished or agreeable. The training process would need to reward encyclopaedic tone, neutral wording, appropriate structure, and detailed compliance with Wikipedia’s style conventions.
1. Supervised fine-tuning
The first stage is supervised fine-tuning (SFT). A base LLM may possess broad linguistic competence and substantial factual coverage, but it will not reliably exhibit the specific editorial discipline required by the WMoS. It may answer fluently while sounding promotional, overconfident, conversational, or insufficiently neutral. It may also ignore conventions around capitalisation, date formats, quotation style, article titles, section structure, or the treatment of disputed terminology.
To address this, we would fine-tune the model on demonstrations of high-quality Wikipedia-style answers and edits. Human demonstrations from expert Wikipedia editors would provide the gold-standard examples, while augmented data could broaden the distribution of training cases after quality control. For instance, editors could write ideal responses to prompts such as:
- “Summarise this disputed topic neutrally.”
- “Revise this paragraph to conform to Wikipedia style.”
- “Convert this promotional biography into an encyclopaedic paragraph.”
- “Rewrite this article lead so that it avoids editorialising.”
- “Format these dates, titles, and quotations according to the Manual of Style.”
Augmented data could include paraphrased prompts, synthetic editing tasks, or retrieval-augmented examples constructed from relevant WMoS passages and then checked by humans.3 For example, the dataset might contain a problematic paragraph, the relevant WMoS guidance, and a corrected version. Beyond imitating the final answer, this would teach the model to associate particular editing decisions with particular rules.
This stage gives the model an initial policy closer to the target behaviour, creating the baseline model whose outputs can then be judged by expert editors. After SFT, the model should be more likely to produce answers exhibiting neutral point of view, concise prose, appropriate hedging, consistent formatting, avoidance of puffery, and respect for constraints such as no original research. However, SFT is primarily imitation learning – meaning it teaches the model to reproduce patterns in demonstrations. It does not, by itself, ensure robust compliance with the WMoS across novel, ambiguous, or adversarial cases.
That limitation matters. Many Wikipedia-style judgements are context-sensitive. Whether a phrase is neutral, whether a biography sounds promotional, or whether an article lead gives undue weight to a controversy cannot always be reduced to a simple rule. This is where human feedback becomes important.
2. Human feedback and reward modelling
The second stage is human feedback collection. Suppose we have access to 100 expert Wikipedia editors who know the WMoS well. We could present them with many prompts and several candidate outputs from the fine-tuned model. The editors would compare or rank these outputs according to both helpfulness and WMoS compliance.
For example, the model might generate three versions of a paragraph about a living public figure. One version may be concise but slightly promotional. Another may be neutral but awkwardly structured. A third may be both neutral and well-structured, but mishandle a date or quotation. Expert editors could rank these answers and annotate the reasons for their preferences. They might prefer one answer because it is more neutral, better organised, less promotional, more precise about dates, or more consistent with Wikipedia’s guidance on biographies of living persons.
Editors could also mark specific defects, such as editorialising, vague attribution, inappropriate capitalisation, poor article-style organisation, excessive detail, or advocacy-like language. These labels would be useful both for preference ranking and for later diagnostic evaluation of where the model still fails to follow the WMoS.
This ranked or paired-comparison preference data would then be used to train a reward model. The reward model learns to predict which outputs the Wikipedia editors would prefer; it is not itself the WMoS, but an approximation of expert editorial judgement.4 This distinction is important. The reward model does not become a perfect rulebook. It learns statistical patterns in human preferences. If the feedback is inconsistent, incomplete, or biased towards superficial signs of Wikipedia-like prose, the reward model may inherit those weaknesses.
Even so, reward modelling is useful because it is usually easier and cheaper for experts to judge whether answer A is better than answer B than to write the ideal answer from scratch for every prompt.5 Once trained, the reward model can score large numbers of candidate outputs automatically, providing a scalable, though imperfect, proxy for the desired objective: helpful answers that satisfy Wikipedia-style editorial constraints.
3. Reinforcement learning against the reward model
The third stage is reinforcement learning. The SFT model generates answers, the reward model scores them, and a reinforcement learning algorithm updates the language model so that high-reward outputs become more likely. In classic RLHF for language models, PPO is often used for this step.6
In this case, high-scoring outputs should be those that expert editors would regard as helpful, neutral, clear, well-structured, and compliant with the WMoS. The model is therefore pushed away from merely fluent answer generation and towards a more specialised editorial style.
A KL penalty is usually added to prevent the policy from drifting too far from the supervised fine-tuned model while optimising for reward. This helps preserve general language ability and reduces the risk of reward hacking, where the model exploits weaknesses in the reward model rather than genuinely improving its editorial behaviour. Without such constraints, a model might learn to produce outputs that look superficially encyclopaedic while still violating substantive rules. It might become very good at sounding like Wikipedia without being good at Wikipedia editing. Immaculate vibes won’t carry poor epistemics.
The expected result is an assistant that is substantially more Wikipedia-compliant than the base model. It should be better at answering in an encyclopaedic register, avoiding loaded phrasing, structuring information appropriately, and respecting Manual of Style conventions. However, RLHF would not guarantee perfect rule-following. The model could still mishandle edge cases, over-optimise for superficial markers of Wikipedia style, or produce polished prose that violates a substantive editorial rule.
For that reason, the system should be evaluated on held-out WMoS tasks, adversarial prompts, and difficult cases involving neutrality, biographies of living persons, disputed terminology, article organisation, quotations, dates, and titles. It would also be useful to test the model on cases where the user explicitly asks for non-Wikipedia-like behaviour: promotional copy, opinionated phrasing, advocacy language, or original synthesis. A good assistant should not merely know the WMoS when convenient; it should maintain the relevant standard under pressure.
Why human editors matter
The 100 expert Wikipedia editors are not just a source of generic approval or disapproval – their expertise defines the target. A random crowdworker might be able to say whether an answer sounds fluent or useful, but they may not reliably detect subtle violations of Wikipedia style. They may prefer confident prose, engaging prose, or rhetorically persuasive prose. Wikipedia editors are more likely to notice the boring-but-important details: neutrality, attribution, scope, formatting, due weight, and the difference between encyclopaedic summary and editorial argument.
RLHF is only as good as the feedback signal – if the feedback rewards the wrong thing, the model will learn the wrong thing efficiently – not a comforting thought, but it is the point. The training process does not remove the need for judgement – in fact it scales whatever judgement is embedded in the data.
Extension: RLAIF
As a further extension, the project could incorporate RLAIF, or Reinforcement Learning from AI Feedback.7 In that setup, an AI system could help generate critiques, preference judgements, or additional training examples, perhaps by retrieving relevant WMoS passages as evidence. This could make feedback cheaper and more scalable.
For example, an AI critic could compare two draft answers, cite relevant WMoS sections, and explain why one answer is more compliant than the other. It could flag likely problems such as loaded language, poor attribution, inconsistent date formatting, or promotional tone. These AI-generated critiques could then be used to expand the feedback dataset.
However, RLAIF should not simply replace expert Wikipedia editors, since the target is expert editorial judgement rather than generic model preference. A more defensible approach would be to use RLAIF to generate draft feedback or candidate examples that human editors then audit, correct, or validate. This would use AI to reduce labour while keeping humans in charge of the normative target.
Conclusion
Training a Wikipedia Manual of Style assistant with RLHF would involve building a pipeline in which expert demonstrations, editor preferences, reward modelling, and reinforcement learning all point towards a specific editorial standard. Supervised fine-tuning would give the model an initial grasp of Wikipedia-style answers. Expert feedback would identify which outputs better satisfy the Manual of Style. A reward model would scale those judgements. Reinforcement learning would then shift the model towards outputs that expert editors would prefer.
The result would still be imperfect. RLHF does not produce guaranteed rule-following, and a reward model is not the same thing as the rulebook it approximates. But with careful dataset design, expert feedback, adversarial evaluation, and human-audited extensions such as RLAIF, it could produce an assistant that is far better aligned with Wikipedia’s editorial norms than a base LLM.
Footnotes
- Wikipedia Manual of Style: https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style ↩︎
- For a widely cited account of RLHF applied to language-model assistants, see Ouyang et al., “Training language models to follow instructions with human feedback” (2022): https://arxiv.org/abs/2203.02155 ↩︎
- Augmented data refers to the use of synthetic generation, modification, paraphrasing, or retrieval-augmented generation (RAG) to expand fine-tuning or preference datasets. For this task, augmented examples should still be checked against the WMoS; otherwise, the model may learn errors at scale. ↩︎
- The usual RLHF process has three broad stages: supervised fine-tuning on demonstrations, training a reward model on human preference comparisons, and using reinforcement learning to optimise the model against that reward model. ↩︎
- The insight behind training a reward model is that it is often easier for humans to judge which of two outputs is better than to write the ideal response from scratch. The reward model learns an approximation of those human judgements so they can be applied automatically during RL training. ↩︎
- PPO stands for Proximal Policy Optimisation, a reinforcement learning algorithm commonly used in classic RLHF pipelines. Related preference-optimisation methods, such as Direct Preference Optimisation (DPO), may be used in some modern post-training workflows, but PPO remains a standard reference point for explaining RLHF. ↩︎
- RLAIF means Reinforcement Learning from AI Feedback. It is related to RLHF, but the feedback is generated partly or wholly by AI systems rather than directly by humans. This is different from merely using augmented data during supervised fine-tuning. For a related approach, see Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (2022): https://arxiv.org/abs/2212.08073 ↩︎
