Preferences, reward models, and direct optimization

A preference record commonly contains a prompt, a chosen response, and a rejected response. It says that one output is preferred under a particular rubric and context. It does not necessarily assign either response an absolute quality score. If both answers are wrong, preferring the less misleading one still does not make it a good demonstration for supervised imitation. Preserve ties, ambiguous cases, and annotator disagreement instead of forcing every comparison into an artificially certain binary label.

Define the rubric before collecting labels. For a technical assistant, correctness, evidence use, authorization compliance, and usefulness may need separate judgments. An annotator who chooses the longer answer because it sounds thorough can teach a length preference rather than better reasoning. Randomize presentation order and inspect agreement on shared calibration examples. Keep prompts and related candidates in the same data partition to avoid contaminating evaluation with memorized comparisons.

The InstructGPT research describes a sequence of supervised demonstrations, human comparisons, reward-model training, and policy optimization. A reward model estimates which outputs people would prefer. A policy then generates responses and is optimized using that learned signal, commonly with a constraint or penalty discouraging excessive drift from a reference policy. The reference helps anchor behavior but does not prove that generated outputs are correct or acceptable.

Trace the errors separately. Human labels may be inconsistent; the reward model may generalize poorly; and policy optimization may discover responses that exploit that model. A held-out reward-model comparison accuracy answers a different question from end-to-end user task success. Maintain evaluations outside the reward-training distribution and review examples where reward rises most quickly. Those examples can reveal useful progress, but they can also reveal a newly discovered shortcut.

Direct Preference Optimization avoids fitting a separate explicit reward model in its basic formulation. Its objective compares the policy's chosen-versus-rejected log-probability gap with the same gap under a fixed reference. If policy log probabilities are -1 and -3, its gap is two. If reference values are -2 and -2, the reference gap is zero. With beta equal to 0.5, the scaled margin is one, and the negative log-sigmoid loss is approximately 0.3133.

If policy and reference gaps match, the margin is zero and the loss is log two, approximately 0.6931. Increasing the chosen response's probability relative to the rejected one lowers this pair's loss. The DPO paper derives the connection to a KL-regularized preference objective; the small code example implements only its scalar loss calculation. It is not a complete trainer, tokenizer pipeline, or proof that an update improves a downstream task.

Sequence log probability is normally a sum over selected response-token log probabilities conditioned on the prompt. Masking prompt tokens and handling response boundaries consistently are therefore essential. Summed probabilities interact with response length, and normalization choices can define a different objective. Do not silently replace a sum with a mean while retaining claims about the original algorithm. Inspect tokenization of chosen and rejected continuations, especially when special tokens or truncation differ.

Beta belongs to both the theoretical regularization relationship and the numerical scale of the training loss. Its practical effect depends on the rest of the setup, so test a documented range rather than treating it as a universal quality knob. A fixed offline preference dataset may not cover errors made by an updated policy. Collecting fresh comparisons can improve coverage, but creates a new data distribution and must preserve a separate evaluation set.

Supervised fine-tuning is attractive when excellent target responses are available. Preference optimization is useful when people can compare candidates more reliably than they can write ideal answers. Online reward-based optimization can explore outputs beyond a fixed dataset, at the cost of additional sampling, infrastructure, and opportunities to exploit a proxy. These approaches can be combined, but each stage should have an explicit hypothesis and ablation.

For a portfolio experiment, start with a fixed candidate set and a clear rubric. Report label counts, disagreement, candidate-generation settings, and the exact comparison unit. Compare a supervised baseline with a preference-trained alternative on held-out prompts using both the rubric and executable checks where possible. Publish examples where the trained model becomes more persuasive without becoming more accurate. Such failures are valuable evidence about the objective's limits and guide whether the next investment should improve labels, evaluation, or the learning method.

The inputs are invented response log probabilities. The code uses a stable expression so extreme margins do not overflow. It demonstrates the loss geometry only; a real update requires model gradients, correct response masks, and a vetted preference dataset.