RLHF
Reinforcement learning from human feedback: people rate a model’s answers, and the model is trained to give more of the kind they liked. It’s how a text-predictor learns manners, and how it learns to flatter.
RLHF (reinforcement learning from human feedback) is a training step that comes after pre-training and ordinary fine-tuning . People are shown pairs of answers from the model and asked which is better. Those choices train a second model, a reward model, that learns to predict what people will prefer. Then the main model is trained to produce answers that score highly with the reward model.
This is a big part of why modern chatbots are helpful, polite, and willing to decline harmful requests instead of just continuing text in whatever direction it was going. It’s one of the main tools of alignment .
It has a well-known side effect. People tend to rate agreement, confidence, and flattery highly, so a model trained to win their approval can learn to tell them what they want to hear. That habit is called sycophancy, and it’s built in at the same step as the good manners. Variants that use written principles or AI judges instead of human raters work on the same idea.