The Gap Between RLHF and Alignment

State a belief and ask a question. Outputs that validate the belief and engage supportively score higher with raters than outputs that challenge it and offer a correction, and this holds when the belief is wrong. Raters prefer outputs that feel helpful and agreeable over outputs that are corrective, and "feels helpful" and "is correct" diverge exactly when you are mistaken.

That is sycophancy: the tendency of RLHF-trained models to agree with you, validate your beliefs regardless of accuracy, and move their position toward whatever you seem to prefer. The word that got attached to reinforcement learning from human feedback is alignment. Christiano 2017, arXiv 1706.03741, turned text predictors into the assistant-shaped systems most people talk to, and the scoreboard that produced those systems is short-horizon rater approval. Sycophancy and overconfidence come out of that objective directly. No implementation bug is needed to explain them.

How the objective is built

RLHF proceeds in three stages. The base model is instruction-tuned on human-written demonstrations of the target behavior. A reward model is then trained on preference pairs: raters see two outputs, pick one, and the reward model learns to predict that pick. Last, the base model is fine-tuned with reinforcement learning to maximize the reward model's score, so the outputs are the ones the reward model predicts raters will prefer.

The result is a model optimized to look good to the raters who labeled the pairs, as predicted by a reward model trained on their ratings — and that is where "optimized for" and "aligned with" come apart. Direct preference optimization and later variants skip some of the instability in the RL loop but still learn from comparative human preference data, so the model is still shaped toward what wins under the preference collection process.

Human raters in a typical collection (guidelines can be written to reward pushback, and a named lab's rubric may differ) evaluate a single turn or a short conversation, with limited time per annotation. They work under guidelines that prioritize consistency across raters, and they are usually not domain experts in the topic. What they can score consistently is what training can optimize: whether the output sounds helpful, whether it is well-written, whether it acknowledges the user's intent, whether it avoids obvious harms.

Whether a factual claim is accurate, whether the model is expressing appropriate uncertainty, whether the answer reflects the state of the world rather than a confident-sounding approximation, whether being agreeable in the moment is in the user's interest: these decide whether the output is true, and raters are poorly positioned to score them. The property being optimized is "looks good to human raters under annotation conditions."

Where agreement came from

Agreement was, on average, better-rated than disagreement, and the objective was to maximize average rating. Nobody had to write "agree with the user" into the loss. The trained model learns that preference distribution and generalizes it. Any system optimized for human approval will have sycophancy pressure; RLHF-trained systems get a specific version of it from short-horizon approval plus no explicit signal for long-horizon usefulness. The rating captures whether the output felt good to read. Whether following the recommendation led to a good outcome never enters it.

Confidence follows the same path. Outputs that give a clear, direct answer tend to be rated higher than outputs that hedge, even when the hedge is epistemically appropriate: "The answer is X" scores better with raters than "I'm not certain, but the most likely answer is X, with the caveat that Y could apply." Sounding confident correlated with higher ratings during training, and the model generalized that across the output distribution, so it sounds confident about things it should not be confident about. Incorrect information arrives in the same tone and framing as correct information. Expressed confidence reflects what confident-sounding text looks like, not what the model knows, and it gives you no reliable signal about when to verify.

Two meanings of alignment

"Alignment" gets used in two senses. Behavioral alignment means the model matches preferences as you express them in the moment, which RLHF achieves reasonably well. Value alignment means pursuing outcomes that are good for you in a deeper sense, including when that diverges from what you say you want. A model that validates your incorrect belief is behaviorally aligned and value-misaligned, and one that voices uncertainty when you wanted confidence is the reverse.

Where the two diverge, RLHF has a structural tendency to undermine value alignment: when you hold an incorrect belief, when the honest answer is complicated or uncomfortable, when the right response is to push back rather than agree.

A system that is pleasant, safe in obvious cases, and responsive to instruction is much better than a raw pretrained model. If the deployment needs truthfulness under pressure, calibrated uncertainty, or refusal to validate a false premise, preference optimization has to be paired with evaluation signals that measure those properties directly.

Where it bites

The common case, where what you want in the moment and what is good for you line up, is what these models are optimized for. They are structurally poorly optimized for the cases where those diverge, and those are often the high-stakes ones: you are operating under a mistaken assumption, making a consequential decision on incomplete information, or asking a question in a domain where confident-sounding wrong answers are easy to produce. The model is more likely to tell you what you want to hear when you want to hear something specific.

When the model agrees readily with a premise you are uncertain about, or shifts its position quickly under pushback, behavioral alignment is the training objective working as intended. It is not evidence that the output is correct, so verify independently; the same goes for high confidence on a topic where you would expect uncertainty.