Design paired experiments and interpret uncertainty
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Separate case variation, training-seed variation, and measurement noise.
- Use paired differences and ablations to test a mechanism.
- Explain the assumptions and limits of a small randomization test.
Control the comparison before increasing its size
A fair comparison holds relevant inputs and resource constraints fixed while changing the proposed intervention. Use the same evaluation cases, documents, allowed tools, and scoring procedure. If one system receives more calls or a stronger model, a higher score cannot isolate the effect of the new orchestration method. Equal resource budgets are not always the only appropriate comparison, but any difference must be explicit and reflected in the research question.
Pair outcomes by case identifier. If baseline and candidate both solve the same easy cases, pairing lets you focus on the cases that changed. With binary outcomes, a table of both correct, baseline only, candidate only, and both wrong reveals the source of an aggregate difference. Preserve missing and failed runs as defined outcomes or report them separately; silently dropping candidate timeouts while keeping baseline failures creates a biased comparison.
Ablation tests the explanation
An ablation removes or replaces a component to see whether the predicted effect depends on it. If a verifier is claimed to catch unsupported answers, compare the full system with no verifier and with a cheap deterministic citation check. A system that improves equally when the verifier's decisions are randomly shuffled suggests that some other effect, such as extra computation or a changed prompt, may explain the result.
Change one interpretable factor at a time when diagnosing mechanisms, while recognizing that interactions can require a factorial design. For two binary components A and B, evaluate neither, A only, B only, and both. If A adds two points alone but ten points when B is present, the interaction deserves analysis. Do not infer that the contribution of the combined system equals the sum of separate ablation gains without checking how the components depend on one another.
Count the right sources of variation
Variation can arise from sampled evaluation cases, model sampling, training seeds, hardware nondeterminism, and human labels. Repeating generation ten times on one question does not give ten independent questions. Training one adapter with five decoding seeds does not reveal variation across training runs. Choose a sampling or resampling unit that matches the claim, and explain whether confidence intervals address case sampling, run variation, or both.
The linked deep-RL reproducibility research demonstrates why nondeterminism and experimental choices can complicate comparisons. In your study, preserve the complete distribution of run results and report the number of independent runs. A fixed random seed improves repeatability of a particular computation, but cannot turn one sampled result into a population guarantee. When compute is limited, narrow the claim and expose the limitation rather than implying that unmeasured variability is absent.
Use a small exact test to understand the null
For paired numerical differences, a sign-flip randomization test compares the observed mean difference with means obtained by independently reversing each pair's sign. Its validity requires the relevant exchangeability or symmetry assumptions under the null; it is not automatically appropriate for every observational comparison. For four pairs with differences [1, 1, 1, 1], there are 16 possible sign assignments. Only all positive and all negative achieve an absolute mean of one, giving a two-sided p-value of 2/16, or 0.125.
All four observed changes favor the candidate, yet this tiny example does not cross a conventional 0.05 threshold. The p-value is not the probability that the candidate is worse, nor the size of its benefit. Report the observed effect and paired outcomes alongside uncertainty. With many comparisons or repeated peeking, selection changes the interpretation, so predefine the primary comparison or use an analysis method that accounts for the study design.
Practical significance and failure slices still matter
A statistically detectable effect can be too small to justify added cost or operational complexity. Conversely, a small study may have uncertainty too large to establish a practically valuable effect. Define a meaningful improvement threshold from the task, then ask whether the evidence distinguishes it from negligible benefit or harm. Include cost, latency, abstention, and consequential errors as separate measures rather than hiding them in an unexplained composite score.
Inspect slices selected before evaluation, such as long contexts, missing evidence, conflicting documents, or tool failures. Treat newly discovered slices as exploratory unless independently confirmed. A candidate that improves easy cases while failing a critical authorization case may be unacceptable despite a positive average. The final conclusion should name the measured difference, the assumptions behind uncertainty, and the conditions under which the result is likely to be useful.
Work through the code
This exhaustive test is feasible only for small n because it evaluates 2 to the n sign assignments. The fixture is invented to show how a p-value is constructed. Real studies must justify exchangeability and choose the correct independent unit before using the test.
from itertools import product
def sign_flip_test(differences):
observed = abs(sum(differences) / len(differences))
extreme = 0
assignments = 0
for signs in product((-1, 1), repeat=len(differences)):
mean = sum(sign * value for sign, value in zip(signs, differences)) / len(differences)
extreme += abs(mean) >= observed - 1e-12
assignments += 1
return observed, extreme / assignments, assignments
differences = [1.0, 1.0, 1.0, 1.0]
effect, p_value, assignments = sign_flip_test(differences)
print(f"absolute mean difference: {effect:.3f}")
print(f"two-sided p-value: {p_value:.3f}")
print(f"sign assignments: {assignments}")
print("assumption: pair signs are exchangeable under the null")
absolute mean difference: 1.000 two-sided p-value: 0.125 sign assignments: 16 assumption: pair signs are exchangeable under the null
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A candidate solves 85 of 100 cases and a baseline solves 80. The candidate fixes eight baseline errors but introduces three new errors. What paired information should be reported, and why is the net gain insufficient on its own?
Check your understanding
A study repeats one prompt with 100 random seeds and claims a confidence interval over all support questions. What is the main problem?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.