Paired tests, ablations, and uncertainty about improvements
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Analyze paired binary outcomes using discordant cases.
- Distinguish statistical evidence from practical importance.
- Design ablations that preserve comparable budgets and datasets.
Pair the outcomes by task
When comparing two agent versions, run both on the same tasks and align their results by stable task ID. This paired design removes one avoidable source of variation: one version should not receive an easier sample merely by chance. Record each task's two outcomes, cost, latency, and relevant failure category. A comparison of two aggregate percentages loses the information about which tasks changed.
For binary success, divide tasks into both succeed, both fail, new-only success, and old-only success. The last two groups are discordant pairs. If the new system gains 12 tasks and loses eight, its net gain is four tasks, even if it fixes several visually impressive examples. Inspect the losses as carefully as the gains. They can reveal that a prompt change improved one task family by damaging another.
Use an exact calculation for a small paired sample
Under a null hypothesis that either system is equally likely to win a discordant task, the number of new-system wins among discordant pairs follows a binomial model with probability one-half, assuming independent task pairs. An exact two-sided McNemar calculation can therefore sum the relevant binomial tails. The example implements this small-sample calculation with the standard library.
In eight invented tasks, the new system wins three and loses one. Four other pairs tie. The observed success difference is two divided by eight, or 0.25. Conditional on four discordant pairs, the two-sided p-value is 0.625. This does not prove that the systems are equivalent. It says the small observed pattern does not provide strong evidence against that null under the stated assumptions. More independent tasks may be needed to estimate the improvement usefully.
Quantify uncertainty without changing its meaning
A confidence interval describes uncertainty from a specified sampling procedure; it is not a probability that one fixed realized interval contains the parameter. A paired bootstrap resamples task pairs together, preserving the within-task relationship, and recomputes a statistic such as mean success difference. If tasks are clustered by document or repository, resample at the cluster level when that clustering is the source of dependence.
Repeated stochastic runs answer another question: how much does the agent vary on the same task? Ten runs of one issue are not ten independent issues. Preserve task identity and use a hierarchical analysis or summarize per-task performance before comparing systems. Report the sample size at each level. Narrow-looking intervals built from dependent repeats can create false confidence about generalization to new tasks.
An ablation tests a causal story
If a new agent adds retrieval, reflection, and a stronger model simultaneously, an overall improvement cannot identify which addition mattered. An ablation removes or changes one component while keeping the comparison as similar as possible. Specify the hypothesis first: perhaps verification reduces incorrect tool arguments, or retrieval improves citation coverage on evidence-heavy tasks.
Control the dataset, tool environment, grading rule, and resource budget where appropriate. Removing a component may free tokens or time; decide whether the ablation preserves total budget or measures the actual deployment policy with its changed spending. Either can be valid, but they answer different questions. A negative ablation result also has limits: a component may interact with another component, so its effect when removed alone need not equal its effect in every architecture.
Choose a decision rule before seeing the result
Practical importance depends on consequences and operating constraints. A tiny statistically detectable gain may not justify a large cost increase, while a promising gain on a small sample may justify a limited follow-up experiment. Define the minimum useful improvement, critical regression limits, and budget constraints before inspecting the test results. If many variants are tried, acknowledge selection effects and reserve fresh evaluation data for the chosen variant.
Publish paired counts, uncertainty methods, costs, and failure examples alongside the headline score. The lab intentionally returns an exact p-value rather than a universal ship-or-reject decision. It is the engineer's job to connect evidence to the product contract. The best report states what improved, how large the change was, how uncertain it remains, and which unresolved risks or task slices motivate the next experiment.
Work through the code
The binary outcomes are invented and paired in the same order. The exact test conditions on discordant pairs and assumes independent task pairs. The p-value is not the probability that the new system is worse or that the null is true.
from math import comb
old = [1, 0, 0, 1, 0, 1, 1, 0]
new = [1, 1, 1, 0, 1, 1, 1, 0]
wins = sum(a == 0 and b == 1 for a, b in zip(old, new))
losses = sum(a == 1 and b == 0 for a, b in zip(old, new))
discordant = wins + losses
def exact_p(wins, losses):
n = wins + losses
if n == 0:
return 1.0
tail = sum(comb(n, k) for k in range(min(wins, losses) + 1))
return min(1.0, 2 * tail / (2 ** n))
delta = (wins - losses) / len(old)
print("wins:", wins, "losses:", losses)
print(f"delta: {delta:.3f}")
print(f"two-sided p: {exact_p(wins, losses):.3f}")
wins: 3 losses: 1 delta: 0.250 two-sided p: 0.625
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A team tests 25 prompt variants on the same 20 tasks, selects the highest score, and reports a p-value against the original version on those same tasks. What is the problem, and what should the next experiment do?
Check your understanding
A new system wins five discordant pairs and loses none. The exact two-sided p-value is 0.0625. Which interpretation is sound?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.