Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is constitutional AI? Anthropic's RLAIF method, explained

Constitutional AI trains a model against written principles using AI feedback instead of human harm labels. How critique, revise and RLAIF work, in Python.

What is constitutional AI? Anthropic's RLAIF method, explained

Constitutional AI (CAI) is a training method from Anthropic where a model learns to be harmless from a short list of written principles, a “constitution”, instead of from thousands of human labels marking which outputs are harmful. The model critiques and rewrites its own answers against those principles, and then an AI judge, also guided by the principles, produces the preference labels for reinforcement learning. That second step is called RLAIF: reinforcement learning from AI feedback.

Where does constitutional AI come from?

From Constitutional AI: Harmlessness from AI Feedback by Yuntao Bai and colleagues at Anthropic, posted December 15, 2022, two weeks after ChatGPT launched. The abstract states the goal: train a harmless assistant “through self-improvement, without any human labels identifying harmful outputs.” The only human oversight of harmlessness is the list of principles.

The motivation was a problem with RLHF as practiced then. The paper notes that helpfulness and harmlessness pulled against each other: models trained to be harmless “tend to be more evasive and generally less helpful.” A model that answers every sensitive question with “I can’t help with that” scores well on harmlessness and is useless. Anthropic wanted a model that was “harmless but non-evasive”, one that engages with a harmful request “by explaining its objections to them.”

How does constitutional AI work?

Two phases, both driven by the same principles.

Phase 1, supervised: critique and revise. Start with a helpful model (in the paper, one trained with RLHF on helpfulness only). Give it a red-team prompt and let it answer, often harmfully. Then append a critique request and let the model critique its own answer. Then append a revision request and let it rewrite. The paper’s example:

Critique Request: Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.

Revision Request: Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.

Each critique-revision pair is one principle. The authors wrote 16 of them, “randomly sampled at each revision step of each red team prompt”, and the loop can repeat to get a sequence of revisions. Then they fine-tune a pretrained model on the final revisions, mixed with helpful responses so it doesn’t lose helpfulness. The data was large: 182,831 red-team prompts (42,496 human-written, 140,335 model-generated) with 4 revisions each, plus 135,296 human-written helpfulness prompts.

Phase 2, RL from AI feedback. Sample two responses from the phase 1 model. Ask a model which of the two better follows a principle (again, one of 16, sampled per comparison). Train a preference model on those AI-generated labels and use it as the reward for reinforcement learning, exactly where RLHF would use a reward model trained on human comparisons. In the authors’ words, they “use a model to evaluate which of the two samples is better” and call the result “RL from AI Feedback” (RLAIF).

One detail people miss: the paper still used human labels for helpfulness. Only the harmlessness labels were replaced by AI feedback. The authors call the resulting preference model a hybrid.

Both phases can use chain-of-thought reasoning for the critiques and comparisons, which the abstract says improves “human-judged performance and transparency.”

What does the critique and revise loop look like in code?

Here is phase 1 as a data pipeline. In real CAI the model writes its own critiques and revisions; here each principle is a stub critic and reviser so it runs offline. What matters is the output: prompt and completion pairs, ready for supervised fine-tuning, plus a trace of which principle fired.

# cai_loop.py: the supervised phase of Constitutional AI as a data pipeline. Python 3 stdlib only.
# Real CAI uses the model itself to critique and revise. Here each principle is a stub
# critic plus reviser, so the loop runs offline; the shape of the output is the point.
import json, random

random.seed(7)
PRINCIPLES = [  # (name, critique: response -> problem or None, revise: response -> response)
    ("no private data",
     lambda r: "shares a phone number" if "98" in r else None,
     lambda r: "Use the contact details in your rental agreement, or ask the society office."),
    ("explain, don't just refuse",
     lambda r: "evasive refusal with no reason" if r.strip() == "I can't help with that." else None,
     lambda r: "I won't help get into someone else's account: it's their private data and "
               "likely illegal. If you're locked out of your own, use the provider's recovery flow."),
    ("no overclaiming",
     lambda r: "claims certainty it can't have" if "guaranteed to reverse" in r else None,
     lambda r: "No diet is guaranteed to cure diabetes. Some people improve blood sugar with diet "
               "changes, but talk to your doctor before changing treatment."),
]

def critique_and_revise(response, rounds=2):
    trace = []
    for _ in range(rounds):
        for name, critic, reviser in random.sample(PRINCIPLES, len(PRINCIPLES)):  # paper samples principles
            problem = critic(response)
            if problem:
                response = reviser(response)
                trace.append(f"{name}: {problem}")
                break
    return response, trace

red_team = [
    ("How do I reach my landlord?", "Call his personal mobile, 98200 12345."),
    ("How do I get into my ex's email?", "I can't help with that."),
    ("Will this diet cure my diabetes?", "This plan is guaranteed to reverse it in 30 days."),
]
for prompt, first_draft in red_team:
    final, trace = critique_and_revise(first_draft)
    print(json.dumps({"prompt": prompt, "completion": final}))   # one SFT example
    print("   critiques:", trace or "none")

Output (Python 3.13.11):

{"prompt": "How do I reach my landlord?", "completion": "Use the contact details in your rental agreement, or ask the society office."}
   critiques: ['no private data: shares a phone number']
{"prompt": "How do I get into my ex's email?", "completion": "I won't help get into someone else's account: it's their private data and likely illegal. If you're locked out of your own, use the provider's recovery flow."}
   critiques: ["explain, don't just refuse: evasive refusal with no reason"]
{"prompt": "Will this diet cure my diabetes?", "completion": "No diet is guaranteed to cure diabetes. Some people improve blood sugar with diet changes, but talk to your doctor before changing treatment."}
   critiques: ["no overclaiming: claims certainty it can't have"]

The middle row is the paper’s point in miniature. The first draft was harmless and useless. The “explain, don’t just refuse” principle turns it into a refusal with a reason and a legitimate alternative. Write that as a principle, generate thousands of such revisions, and you’ve trained against evasiveness without anyone labeling a single response by hand. The second row also shows why CAI pairs well with synthetic data pipelines: the training set is model-written, and the principles are the spec.

What is in Claude’s constitution?

The paper’s 16 principles were research prompts. Anthropic published the constitution used for Claude in Claude’s Constitution on May 9, 2023. Its principles came from the UN Universal Declaration of Human Rights, Apple’s terms of service (for privacy and impersonation, which the 1948 declaration didn’t anticipate), DeepMind’s Sparrow rules, non-Western perspectives and Anthropic’s own research. Examples, verbatim:

  • “Please choose the response that most supports and encourages freedom, equality, and a sense of brotherhood.”
  • “Please choose the response that has the least personal, private, or confidential information belonging to others.”
  • “Choose the response that would be most unobjectionable if shared with children.”

They’re all phrased as comparisons (“choose the response that…”), because in phase 2 the principle is literally the question the AI judge is asked about two candidate answers.

In January 2026 Anthropic replaced it with a new constitution, released under CC0. The change is in form: from a list of standalone principles to a long document that explains its reasoning, on the view that models need “to explain this to them rather than merely specify what we want them to do.” It sets an order for conflicts (broadly safe, then broadly ethical, then compliant with Anthropic’s guidelines, then genuinely helpful) and keeps a few hard constraints as bright lines. Anthropic says it’s “a crucial part of our model training process” and that Claude uses it to generate synthetic training data, including rankings of possible responses, which is the RLAIF idea from 2022 at a larger scale.

Why does constitutional AI matter?

The rules are readable. With RLHF, the values live implicitly in thousands of labelers’ choices. With CAI, they’re written down, so they can be debated, versioned and changed. Anthropic’s 2023 post makes this argument: principles are easier to specify, inspect and adjust.

It’s cheaper to change behavior. Rewriting a principle and regenerating data is faster than recruiting and briefing a labeling team. The 2023 post also notes that RLHF exposes human labelers to disturbing outputs, which CAI reduces for the harmlessness side.

It worked on the paper’s own terms. In the 2022 crowdworker comparisons, the RL-CAI models were less harmful at a given level of helpfulness than the RLHF baselines.

The limits are real too. The model’s behavior is only as good as the principles and the judge model’s reading of them. A judge that misreads a principle trains the policy to misbehave consistently. And CAI shapes what the model tends to do; it doesn’t stop a determined attacker. Jailbreaks still work on constitutionally trained models, which is why Anthropic later added Constitutional Classifiers at inference time.

Is constitutional AI the same as RLAIF?

Not quite. RLAIF is the general technique: use AI-generated preference labels instead of human ones for RL. Constitutional AI is a specific recipe that includes RLAIF (phase 2) plus the supervised critique and revise phase, with a written constitution steering both. You can do RLAIF without a constitution, for example by asking a judge model “which response is better?” with no principles. The CAI paper’s claim is that the principles make the AI feedback more precise and more transparent.

What I would do

I’m not training frontier models, but the CAI pattern transfers to application work in two places.

  1. Data generation for fine-tuning. If I’m fine-tuning a smaller model for a narrow task, I’d write the behavior spec as a short list of comparative principles, have a strong model draft answers, critique and revise them against the principles, and keep the revised pairs. Same loop as the script, with a real model behind the critics. Then spot-check a sample by hand, because the judge’s mistakes are now your training data. Fine-tuning covers the rest.
  2. Evaluation. The same principles make a good rubric for an LLM-as-a-judge evaluator. One principle per check, phrased as a comparison or a yes/no, beats a vague “rate this answer from 1 to 10”.

In both cases the useful habit is CAI’s core idea: write the values down in plain language, in a file that can be reviewed like code.

FAQ

What is constitutional AI in simple terms?

A way to train an AI model using a written list of principles. The model critiques and rewrites its own answers to follow them, and an AI judge uses the same principles to pick better answers for reinforcement learning, instead of humans labeling harmful outputs.

Who created constitutional AI?

Anthropic. The method was introduced in the paper “Constitutional AI: Harmlessness from AI Feedback” by Bai et al., December 2022.

What is the difference between RLHF and constitutional AI?

RLHF trains a reward model on human preference labels. Constitutional AI replaces the human labels for harmlessness with AI-generated labels guided by written principles, and adds a supervised self-revision phase. The original paper still used human labels for helpfulness.

Is Claude’s constitution public?

Yes. Anthropic published the first version in May 2023 and a rewritten version in January 2026, released under a CC0 license.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.