Patch verification from failing behavior to reviewable evidence
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Construct a regression check from the requested contract.
- Separate syntax checks, focused tests, and broader regression gates.
- Report exactly what changed and what verification establishes.
A plausible diff is only a hypothesis
Generated code can look idiomatic while solving the wrong problem. Treat a patch as a proposed explanation of the bug, and choose checks that can falsify it. First establish the baseline behavior in a controlled environment. If a reported regression cannot be reproduced, investigate inputs, versions, and setup before changing logic. A failing setup command is not the same as a failing application test.
Use a concrete contract for the regression case. Suppose count_at_least should include values equal to a threshold, but the implementation uses a strict greater-than comparison. For values four, five, and six at threshold five, the required result is two and the existing result is one. This single boundary case explains the intended edit and gives reviewers a precise reason for changing the operator.
Choose independent expectations
A useful regression test derives its expected value from the task contract, not by reproducing the same expression used in the implementation. If both test and code make the same assumption, they can agree while remaining wrong. Include neighboring behavior that should stay intact: no matching values, all matching values, duplicates at the boundary, and an empty input where the API defines it.
Testing every trivial formatting edit is unnecessary, but behavioral changes deserve checks proportional to their risk. A test that fails before the fix and passes after it provides strong local evidence. It still does not prove the entire program correct. The example runs a small fixed set of trusted cases against two literal implementations, so the relationship between the bug and the fix is visible without executing arbitrary repository code.
Verify in layers and inspect what actually changed
Syntax compilation catches malformed code but not wrong behavior. Focused tests exercise the changed contract. Relevant integration tests inspect boundaries that a local unit test may miss. The repository's required broader gates can catch regressions elsewhere. Select these layers based on the change and its dependencies, and explain any gate that could not run rather than declaring it passed.
Inspect the diff for unrelated edits, generated files, accidental deletions, and changed test expectations. Git apply --check can validate whether a patch applies under its documented conditions; it does not establish semantic correctness. Run checks against the resulting working tree, then inspect it again if a formatter or generator can modify files. A successful test run before the final code change is stale evidence for the final patch.
Control the execution environment
Repository tests execute code and may launch subprocesses, read credentials, or contact services. Use the containment and permission design from the previous module. For controlled subprocess calls, pass an argument list, set the working directory explicitly, and use bounded execution. Python's subprocess interface exposes return codes and captured output, but a timeout does not automatically establish that every descendant process or remote effect was terminated.
Record the runtime, dependency version or lockfile state, test command, exit status, and relevant output. Avoid copying secrets from logs into the final report. If a test is flaky, preserve the first failure and investigate its cause; repeated reruns until green do not explain the failure. Distinguish a reproducible product regression from environmental instability and document which conclusion the available evidence supports.
Close with evidence a reviewer can use
A reviewable handoff leads with the concrete behavior change, then explains why the patch addresses it and how it was checked. Include the focused regression, required gates, and material limitations. Do not claim all tests passed when only one function was compiled, and do not bury a blocked integration suite behind a broad statement of success.
For a portfolio artifact, include the original reproduction, the minimal patch, the verification transcript, and a short analysis of alternatives. Demonstrate that the patch preserves surrounding contracts rather than celebrating the number of files edited. The seed project adds a behavior check to a small source-analysis report. It remains a controlled simulation; extending it to real repositories requires isolated execution, reliable patch application, and the actual project's test and review requirements. The result should help a reviewer decide whether to trust the change and what evidence is still missing.
Work through the code
Both sources are trusted fixed lesson strings. The checks derive expectations from the inclusive threshold contract. exec is used only to compare those literals; do not run arbitrary model or repository code this way without an independent execution boundary.
before = "def count_at_least(values, threshold):\n return sum(v > threshold for v in values)\n"
after = "def count_at_least(values, threshold):\n return sum(v >= threshold for v in values)\n"
cases = [
("boundary", [4, 5, 6], 5, 2),
("empty", [], 5, 0),
("duplicates", [5, 5], 5, 2),
("none", [1, 2], 5, 0),
]
def verify(trusted_source):
namespace = {}
compiled = compile(trusted_source, "<trusted-lesson>", "exec")
exec(compiled, namespace)
failed = []
for name, values, threshold, expected in cases:
actual = namespace["count_at_least"](values, threshold)
if actual != expected:
failed.append(name)
return failed
print("before failures:", verify(before))
print("after failures:", verify(after))
print("checked cases:", len(cases))
before failures: ['boundary', 'duplicates'] after failures: [] checked cases: 4
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
An agent modifies a function and its test so both use the same new helper to calculate the expected result. The test passes. What concern should a reviewer raise, and what independent check would help?
Check your understanding
A patch compiles and git apply --check succeeds. What is established?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.