Evaluate retrieval, grounding, and cost as separate stages

An evaluation needs a fixed unit: query, retrieved passage, generated claim, or completed task. Retrieval quality and answer quality are related but different. A system can retrieve the correct passage and generate an unsupported answer; it can also answer correctly from prior knowledge while failing the intended retrieval test. Measure stages separately so improvement work targets the failing mechanism.

Create a corpus snapshot and labeled questions with answer-bearing passage IDs, required relations, relevant time, access context, and expected answerability. Include easy single-passage questions and harder cases requiring an exception, a graph join, a historical snapshot, or a missing fact. BEIR demonstrates the importance of evaluation across heterogeneous retrieval tasks; the lesson here is to avoid treating one convenient corpus as evidence of universal retrieval quality.

Precision at k divides relevant retrieved items in the first k positions by k. Recall at k divides retrieved relevant items by the number of known relevant items. Reciprocal rank is one divided by the position of the first relevant item, or zero when none is found. These metrics emphasize different aspects: precision limits wasted context, recall checks coverage, and reciprocal rank rewards finding useful evidence early.

For ranking [D2, D1, D3] and relevant set {D1, D4}, precision at three is one-third, recall at three is one-half, and reciprocal rank is one-half. If fewer than k results are returned, document whether precision uses k or the returned count. The code uses k. Deduplicate document IDs before scoring, and decide whether relevance refers to chunks or parent documents so overlapping chunks do not inflate apparent success.

When passages differ in usefulness, normalized discounted cumulative gain can compare a ranking against an ideal ranking using graded relevance and rank discounting. State the gain function, such as 2^relevance - 1, because different conventions change values. Use labels that distinguish a directly answer-bearing passage from a merely topical one. A graph path may need every edge to be supported, so scoring individual retrieved nodes is insufficient.

Evaluate generated answers for claim support, factual correctness, citation accuracy, completeness, and appropriate abstention. Ragas proposes automated evaluation signals for retrieval-augmented generation. Treat such judges as instruments with error, not an unquestionable ground truth. A model judge can miss subtle version mismatches or prefer a fluent unsupported answer. Calibrate it against a human-reviewed sample and retain the source evidence needed to inspect disagreements.

Compare pipelines on the same query set, corpus revision, access context, and resource budget. Change one stage at a time when diagnosing a regression. If a graph system gets a larger context window and more retrieval calls than a lexical baseline, the result mixes architecture with compute. Report latency, index build cost, query cost, and storage alongside quality so the tradeoff is visible.

Use a held-out test set after choosing thresholds on development questions. Do not repeatedly inspect the test failures and tune until the same test set becomes training data. For small portfolios, show paired per-query differences and uncertainty rather than claiming a universal improvement from a handful of examples. Record actual measurements only; a planned benchmark table should remain empty or clearly labeled until the experiment runs.

Tag failures by ingestion loss, chunk boundary, access filter, candidate recall, reranking, temporal mismatch, unsupported inference, or citation mismatch. A recall failure suggests a retrieval change; a correct candidate followed by a wrong answer suggests grounding or generation work. Include unanswerable and denied queries so a system cannot improve apparent completeness by inventing answers or crossing access boundaries.

The code computes three metrics for one synthetic ranking with a declared relevance set. These numbers are arithmetic examples, not measured model performance. The project expands this into a reproducible report over real learner-created fixtures. Its strongest evidence is a trace from failed question to responsible stage and verified repair, supported by a held-out comparison that makes both improvements and remaining limitations visible.

The fixed ranking contains one of two relevant documents, first seen at rank two. Precision uses the requested k as denominator. Empty relevance sets receive zero by this explicit teaching convention and should be reported as a separate unanswerable slice in a real evaluation.