Group-relative learning and reward-hacking diagnostics

Group Relative Policy Optimization was introduced in DeepSeekMath as a variant of PPO that estimates advantages from groups of sampled responses rather than a separately trained value model in that formulation. Several completions are sampled for the same prompt, scored, and compared within their group. A common illustrative advantage subtracts the group's mean reward and divides by its standard deviation. The actual policy objective also contains probability ratios, clipping, and regularization choices; normalization alone is not GRPO training.

For invented rewards [0, 1, 1, 2], the mean is one and the population standard deviation is the square root of 0.5. The normalized advantages are approximately [-1.414, 0, 0, 1.414]. The best response receives positive relative credit and the worst negative relative credit. The group average is zero, so the signal describes relative standing rather than an absolute statement that the task has been solved.

If every completion receives zero, a normalized group has no preference direction from that reward. If every completion receives one, the same problem occurs even though every completion may be acceptable. An implementation must define zero-variance behavior explicitly; returning zero advantages avoids division by zero in a simple simulation. Adding a small epsilon stabilizes a denominator but cannot invent information that the reward does not contain.

Group composition therefore matters. Very hard prompts may produce only failures, while easy prompts may produce only successes. Increasing the number of sampled candidates can reveal differences, but also costs more generation. A richer verifier may provide more useful distinctions, provided its intermediate scores correspond to desired behavior. Do not reward arbitrary formatting complexity merely to create variation. Inspect the distribution of group variance and the fraction of groups that supply a nonzero learning signal.

Policy-gradient methods compare probabilities under the current policy with those used to sample the data. Large ratios indicate that optimization is moving away from the sampling policy; PPO-style clipping limits the incentive for some oversized updates. A reference-related penalty can discourage unwanted drift. These mechanisms manage optimization behavior and do not certify that a reward is aligned with the intended outcome.

Suppose a verifier awards one point whenever the final line contains the expected numeric string. A response that prints every likely number might collect reward without solving the problem. Conservative updates can still move toward that exploit if it repeatedly scores well. The remedy belongs in task definition, verifier design, independent evaluation, and action boundaries. Stability of the loss curve is valuable operational evidence, but it is not evidence that the underlying objective measures the right thing.

Create candidate outputs that deliberately separate the proxy from the real goal. For a citation task, include a fluent answer with invented citations, an accurate answer with a missing citation, and a response that copies citation syntax without supporting the claim. For a coding task, test solutions that special-case visible examples, suppress errors, or modify the test interface. Run this work only against your own fixtures and authorized evaluation environment.

Score each candidate with both the training reward and an independent criterion. A disagreement table can reveal precisely what optimization might exploit. Reserve hidden test cases, inspect unusually high rewards, and limit the evaluator's access so a generated candidate cannot rewrite its judge or read protected answers. If the judge is another model, treat it as an imperfect measurement device with its own sensitivity to verbosity, ordering, and adversarial instructions.

An honest training report separates sampled reward, held-out reward, executable task success, behavior regressions, and resource cost. A rising training reward with flat independent success is an alarm worth investigating, not a result to average away. Track outcome changes for individual prompts and failure categories. Also measure output length and repeated patterns: reward shortcuts often change these distributions before aggregate correctness visibly deteriorates.

A useful first experiment is an offline reward audit rather than a large reinforcement-learning run. Assemble a small, varied candidate corpus, document why each candidate should succeed or fail, and evaluate how well the proposed reward ranks it. Then compare simple reward variants under a fixed candidate budget. If a reward cannot distinguish an obvious exploit in this controlled setting, allocating more policy optimization to it is unlikely to solve the measurement problem.

This deterministic function illustrates one group-normalization convention using population variance. It is not a GRPO implementation: it does not sample a model, compute token probabilities, apply policy ratios, or update parameters.