From product contracts to representative evaluation data
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Translate a user task into outcome and process metrics.
- Design evaluation cases with provenance and meaningful slices.
- Separate abstention, correctness, and consequential errors.
Evaluate the contract the user cares about
An agent can produce fluent text while failing the task. A report may omit a required source, a booking may remain incomplete, or a patch may pass syntax checks while preserving the bug. Begin with an acceptance contract that describes observable success. Include the final artifact, required evidence, tool effects, and constraints such as permission scope. The evaluator should inspect those outcomes rather than rewarding a convincing explanation of success.
Separate final success from diagnostic process measures. Retrieval recall, schema validity, tool-call count, and self-correction rate help locate failure mechanisms, but none alone establishes that the user received a correct result. If a task requires both a correct answer and no unauthorized action, report both conditions and their conjunction. A high answer score must not conceal a failed action constraint.
Specify the denominator before the metric
Suppose six tasks yield four correct answers, one incorrect answer, and one explicit abstention. Overall task accuracy is four divided by six. Coverage is five divided by six because five tasks received an answer. Accuracy among answered tasks is four divided by five. An agent that abstains on every difficult task can improve conditional accuracy while becoming less useful. Report coverage together with selective accuracy.
Add separate categories for timeout, infrastructure error, invalid output, and grader uncertainty. Decide before evaluation whether these count as unsuccessful tasks for the primary metric, and preserve the diagnostic category even if they share that primary score. Do not quietly remove inconvenient rows after a run. A denominator that changes with model behavior can make two systems appear comparable when they were evaluated on different effective task sets.
Build cases from real variation
A useful dataset includes ordinary tasks and the variations that affect decisions: short and long context, missing evidence, conflicting sources, tool failures, ambiguous requests, and permission boundaries. Sample production-like requests only through authorized data handling. Synthetic cases can target failure mechanisms, but they should be labeled and should not be mistaken for a representative traffic sample.
Each case needs a stable ID, task text, environment or fixture version, expected outcome, grading rule, provenance, and slice labels. Datasheets for Datasets motivates documenting how data was created and should be used. For this course's report assistant, an original case record might specify an immutable source revision, required citation IDs, and whether an answer is possible. That record allows a future engineer to reproduce why an abstention was correct instead of guessing from a bare answer string.
Prevent leakage through grouping and iteration
Split related examples together. If ten questions are paraphrases of the same source document, randomly placing them across development and test sets leaks document-specific patterns. Group by customer, document family, repository issue lineage, or time period when those relationships would create overlap. Use development cases to improve prompts and reserve a test set for a predeclared comparison.
Treat graders as software that needs validation. An exact string check is appropriate for a normalized identifier but can reject equivalent prose. A model grader needs a clear rubric, representative human-reviewed examples, and checks for order or style bias. Keep grader instructions and reference answers away from the agent under test. If a model can inspect its grader or alter the tests, the experiment may measure access to the answer rather than task capability.
Use behavioral tests to explain failures
Aggregate accuracy says how often a system passed on a dataset; targeted tests help explain what it can handle. CheckList is a primary research example of behavioral testing beyond a single aggregate score. For an agent, create invariance tests where irrelevant formatting changes should not alter the answer, and directional tests where changing a cited amount must change the computed total.
Pair each targeted transformation with a reason. Replacing a report title should not alter its arithmetic, but replacing a currency may require conversion or abstention. A metamorphic test is useful only when the expected relationship is valid. The code below computes coverage and accuracy on six invented cases. It is an evaluation accounting example, not evidence about any real model. Build richer fixtures and verify their labels before drawing conclusions from a larger number of runs.
What does your metric hide?
Increase false negatives while holding the other counts fixed. Precision stays unchanged, even though more relevant cases are missed. Decide which failure is costly before selecting a metric.
Work through the code
Six synthetic records show why coverage and conditional accuracy need separate denominators. None represents a real model run. The function assumes a nonempty dataset and non-null reference answers; extend the contract explicitly before handling empty or unanswerable cases.
cases = [
{"id": "a", "expected": "A", "answer": "A"},
{"id": "b", "expected": "B", "answer": "B"},
{"id": "c", "expected": "C", "answer": "D"},
{"id": "d", "expected": "A", "answer": None},
{"id": "e", "expected": "B", "answer": "B"},
{"id": "f", "expected": "C", "answer": "C"},
]
def summarize(records):
answered = [r for r in records if r["answer"] is not None]
correct = sum(r["answer"] == r["expected"] for r in answered)
return {
"coverage": len(answered) / len(records),
"overall": correct / len(records),
"answered_accuracy": correct / len(answered) if answered else None,
}
for name, value in summarize(cases).items():
print(f"{name}: {value:.3f}")
coverage: 0.833 overall: 0.667 answered_accuracy: 0.800
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A new agent answers 60 of 100 tasks and gets 57 right. The old agent answers 90 and gets 72 right. Compare coverage, answered accuracy, and overall correctness without declaring a universal winner.
Check your understanding
Ten questions are paraphrases grounded in one confidential report. How should a held-out split usually treat them?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.