Supervised fine-tuning is a weighted prediction problem
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Compute masked token cross-entropy.
- Reason about length weighting and effective batch size.
- Design a small adaptation experiment with a meaningful holdout.
What the optimizer actually sees
Supervised fine-tuning adjusts parameters to increase the likelihood of demonstrated continuations. In a causal language model, each target token is predicted from earlier tokens. Teacher forcing supplies the true previous tokens during training, even though deployment feeds generated tokens back into the next step. Consequently, a low training loss does not by itself show that a long autonomous interaction will remain on track after an early mistake.
Build each example with the same role boundaries, tool-result structure, and special-token conventions that inference expects. The demonstration must contain information the deployed model will actually receive. A training answer based on an internal field omitted at inference is an impossible supervision target. Before spending accelerator time, decode a few fully formatted batches and inspect the exact tokens and roles that the optimizer will learn from.
Masks define the task
A loss mask decides which positions contribute to the objective. In assistant-only training, user and system tokens provide context but are not prediction targets for the selected loss. This is different from removing those tokens: the assistant still needs to condition on the question. Current TRL documentation describes assistant-only loss for compatible conversational templates; the template must be capable of exposing assistant-token spans. Verify behavior on your pinned software version instead of assuming that a flag alone proves correctness.
Suppose the probabilities assigned to two desired assistant tokens are 0.8 and 0.5. Their negative log likelihoods are approximately 0.2231 and 0.6931. Averaging gives 0.4581 nats per supervised token. The corresponding perplexity is exp(0.4581), about 1.5811. A padding token or user token with a poor predicted probability should not change that assistant-only loss when its mask is zero.
Length weighting is an implicit policy
Averaging all supervised tokens gives long answers more total influence. If one example contributes two tokens and another contributes eighteen, the second supplies 90 percent of the terms in a token-weighted batch objective. Averaging each example first and then averaging examples changes the weighting. Neither choice is automatically correct: a task dominated by concise structured responses may need different sampling than one teaching long technical explanations.
Padding and packing address utilization rather than data quality. Padding creates equal tensor shapes and requires excluded positions in the loss. Packing places multiple examples into available sequence capacity, but the implementation must preserve intended boundaries, attention behavior, and masks. Truncating every sequence to a fixed limit can systematically remove final answers or tool outcomes. Measure supervised-token coverage and truncation by task type before interpreting a training curve.
Budget in tokens and updates
Effective batch size depends on devices, examples per device, and gradient accumulation, but examples with different lengths make token counts more informative. Four devices processing two examples each with eight accumulation steps produce 64 examples per optimizer update. If the average supervised length is 200, that is roughly 12,800 supervised tokens per update. It is only an estimate because padding, variable lengths, and sampling change the actual count.
Accumulation sums gradients over microbatches before an update. It reduces peak activation memory relative to processing the whole effective batch at once, but does not make all large-batch dynamics identical, especially with different normalization or stochastic behavior. Track optimizer steps, tokens seen, learning rate, and wall time. A claimed cheaper run should identify whether it processed fewer tokens, updated fewer parameters, used different hardware, or changed the training objective.
A small experiment can still answer a real question
Choose one observed behavioral failure before adapting: perhaps invalid JSON, missed escalation, or inconsistent use of supplied evidence. Compare a frozen baseline, a prompt-only improvement, and a small supervised adapter using the same protected evaluation cases. Include a general-capability check because specialization can degrade behaviors outside the adaptation data. Keep decoding settings fixed when comparing outputs, and examine paired case changes rather than only aggregate means.
Watch for training loss falling while held-out task performance stalls. Possible causes include memorization, mismatched formatting, label noise, or optimizing fluent imitation instead of the actual success criterion. An early stopping rule should be chosen on development data, with the final holdout used sparingly. Publish the data manifest, configuration, and failure examples alongside scores. A modest, reproducible improvement on a narrow task is stronger evidence than an unexplained large number.
Work through the code
The probabilities stand in for model outputs; no neural network is trained. The mask selects assistant targets while leaving other positions out of the average. Change probabilities only at masked-out positions to confirm that their values do not affect the reported loss.
import math
def masked_nll(probabilities, mask):
if len(probabilities) != len(mask):
raise ValueError("length mismatch")
selected = []
for probability, include in zip(probabilities, mask):
if not 0 < probability <= 1:
raise ValueError("invalid probability")
if include:
selected.append(-math.log(probability))
if not selected:
raise ValueError("no supervised tokens")
return sum(selected) / len(selected), len(selected)
probabilities = [0.01, 0.2, 0.8, 0.5, 0.001]
mask = [False, False, True, True, False]
loss, count = masked_nll(probabilities, mask)
print(f"supervised tokens: {count}")
print(f"loss: {loss:.6f}")
print(f"perplexity: {math.exp(loss):.6f}")
supervised tokens: 2 loss: 0.458145 perplexity: 1.581139
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
Example A has two supervised tokens with mean loss 0.2. Example B has eight with mean loss 0.8. Compute the token-weighted and example-weighted means, then describe the behavior each objective emphasizes.
Check your understanding
Training loss falls sharply, but invalid JSON on a protected evaluation set is unchanged. What is the best next action?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.