What Is RLHF? Reinforcement Learning from Human Feedback, Step by Step

Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns a foundation model’s behavior with human preferences and values. Instead of teaching the model new facts, RLHF teaches it how people want it to respond — more helpful, more honest, less harmful — by having human reviewers rank candidate outputs, training a reward model to predict those rankings, and then using reinforcement learning to steer the foundation model toward responses the reward model scores highly. It is the standard answer to a hard problem: “be helpful and safe” cannot be written as a loss function, but humans recognize it when they see it.

Why alignment needs human feedback

Pre-training optimizes a model to predict the next token over enormous text corpora. That produces fluency and knowledge, but the internet’s most statistically likely continuation is not necessarily the most useful, truthful, or appropriate one. A purely pre-trained model will happily produce evasive, verbose, biased, or unsafe text when the data distribution points that way.

Supervised fine-tuning helps — you can show the model examples of good responses — but writing perfect demonstrations for every situation doesn’t scale, and demonstrations can’t express relative judgments. Humans are much better at comparing than authoring: shown two candidate answers, a reviewer can quickly say which is clearer, safer, or more helpful, even when neither is perfect and neither reviewer could have written an ideal answer from scratch. RLHF is built around exactly that comparative signal. Its primary purpose, in exam terms: to refine a model’s behavior so its outputs align with human preferences and values — not to add knowledge, not to shrink the model, not to speed up inference.

The RLHF pipeline, in order

The workflow has a fixed sequence, and the exam cares about the order.

Step 1 — Start from a capable base. RLHF assumes a pre-trained (and usually supervised-fine-tuned) foundation model that can already produce reasonable responses. RLHF shapes behavior; it doesn’t create competence. How that base gets built is covered in the foundation model lifecycle.

Step 2 — Collect human preference data. The model generates multiple candidate responses to the same prompt. Human reviewers compare the candidates and rank them from best to worst against criteria like helpfulness, accuracy, and safety. Repeating this across thousands of prompts yields a dataset of human preference rankings. On AWS, Amazon SageMaker Ground Truth provides the human-workforce tooling for gathering this kind of feedback at scale.

Step 3 — Train a reward model. The rankings are used to train a separate model — the reward model — that takes a prompt and a response and outputs a score predicting how a human reviewer would rate it. This is the pivotal, most-tested step: after the rankings are collected, the next thing that happens is training the reward model, not updating the foundation model. The reward model converts sparse, expensive human judgments into an automatic scorer that can evaluate unlimited responses for free.

Step 4 — Optimize the foundation model with reinforcement learning. Only now does the foundation model’s behavior actually change. A reinforcement learning algorithm — commonly Proximal Policy Optimization (PPO) — has the model generate responses, gets them scored by the reward model, and adjusts the model’s weights to make high-scoring responses more likely. A constraint keeps the updated model from drifting too far from the original, which protects fluency and guards against the model gaming the reward signal with degenerate outputs.

The two human-touching and model-touching halves are worth separating in your head: humans rank outputs and (indirectly) train the reward model; reinforcement learning then uses that reward model to update the foundation model. Any answer choice that has humans hand-editing the model’s weights, or has the model updating itself directly from raw rankings with no reward model in between, is describing something other than standard RLHF.

RLHF vs supervised fine-tuning

Both techniques adjust a model after pre-training, and the exam likes to check that you can tell them apart.

	Supervised fine-tuning (SFT)	RLHF
Training signal	Explicit example outputs (demonstrations)	Human preference rankings via a reward model
What humans provide	The correct response itself	Comparative judgments between candidate responses
Optimization method	Standard supervised learning on input-output pairs	Reinforcement learning (e.g., PPO) against a reward score
Best at	Teaching format, domain style, specific tasks	Aligning tone, helpfulness, and safety with human values
Typical position	Before RLHF in the pipeline	Final behavioral refinement stage

In practice they are stages of one pipeline, not rivals: pre-train, supervise-fine-tune on demonstrations (which is where fine-tuning data preparation earns its keep), then apply RLHF to polish alignment. Note also what RLHF is not: it is unrelated to preparing a training corpus (that’s data preparation), to grounding responses in external documents (that’s Retrieval-Augmented Generation), and to runtime content filtering (on AWS, that’s Guardrails for Amazon Bedrock — a deploy-time safety layer, whereas RLHF bakes preferences into the weights during training).

How the AIF-C01 exam tests this

• Purpose questions. “What is the primary purpose of applying RLHF?” The credited answer is aligning model behavior/outputs with human preferences and values. Distractors offer plausible-sounding goals — reducing model size, accelerating inference, expanding factual knowledge, compressing training data — that RLHF does not serve.
• Sequence questions. Given that human reviewers have just ranked candidate responses, what happens next? The reward model is trained on those rankings before any reinforcement learning updates the foundation model. Answers that jump straight from rankings to model updates skip the load-bearing step.
• Pipeline-component questions. Choose-two items ask which actions belong to a standard RLHF pipeline. Valid picks: humans ranking/comparing candidate outputs, training a reward model on preference data, RL optimization against the reward model. Invalid picks: unsupervised pre-training on web text, manual weight editing, prompt engineering, output filtering.

RLHF questions belong to the Applications of Foundation Models domain — the full AIF-C01 study guide covers how to budget study time across it and the other domains, and the pipeline order sticks fastest when you drill it with practice questions.

Quick reference

• RLHF = Reinforcement Learning from Human Feedback: aligns model behavior with human preferences and values.
• It shapes how a model responds (helpful, honest, harmless); it does not add knowledge or reduce cost.
• Pipeline order: capable base model → humans rank candidate responses → train reward model on rankings → RL (e.g., PPO) updates the foundation model.
• The reward model is the bridge: it turns limited human rankings into an unlimited automatic scoring signal.
• Humans never directly edit weights; reinforcement learning applies the changes using reward-model scores.
• SFT teaches from demonstrations; RLHF teaches from comparisons — SFT usually runs first, RLHF second.
• Amazon SageMaker Ground Truth supports collecting human feedback; Guardrails for Amazon Bedrock is runtime filtering, not RLHF.