Combine lexical precision with semantic recall

Sparse retrieval represents documents through terms and associated weights. BM25 combines term rarity, a saturating contribution from repeated terms, and document-length normalization. This makes a rare exact identifier useful without letting endless repetition increase its contribution linearly. Tokenization, case handling, stemming, and field structure still influence which matches exist. The ranking formula cannot repair an identifier destroyed during ingestion.

For a query about error E742, exact lexical matching can be more useful than broad semantic similarity. A dense retriever might return documents about similar errors while missing the exact code. Keep an explicit evaluation slice for identifiers, product names, versions, and quoted phrases. These cases often reveal why replacing a good lexical baseline with a more elaborate model can reduce practical reliability.

Dense Passage Retrieval encodes questions and passages into vectors whose similarity supports retrieval. Bi-encoder designs allow document representations to be computed ahead of time, which makes large candidate searches practical. A query about requests waiting too long may retrieve a passage using timeout configuration even without exact word overlap. This is useful semantic recall, but the similarity value is not the probability that the passage answers the question correctly.

Dense representations can blur distinctions involving negation, dates, version numbers, or neighboring entities. Embedding choice, training domain, vector normalization, and similarity metric affect results. Record the embedding model and index version together. Mixing query vectors from one model with stored vectors from another generally breaks the intended geometry even if both happen to have the same dimensionality.

The code indexes three short documents. One contains timeout configuration, another billing payment, and the third configuration guide. The query asks for timeout configuration. The first document matches both terms, the third matches one, and the billing document matches neither. A positive-IDF BM25 variant produces that expected order while accounting for term counts and document lengths.

With parameters k1 equal to 1.2 and b equal to 0.75, increasing a term's frequency from one to two helps less than doubling a linear count would. Longer documents receive a length adjustment relative to average length. These are tuning parameters, not universal constants. The implementation is intentionally small: it omits fields, token positions, language analysis, and an inverted index, making it suitable for inspection rather than production-scale search.

A BM25 score of 7 and a cosine score of 0.8 are not naturally comparable. Reciprocal rank fusion combines positions instead: each list contributes 1 / (k + rank) for a document it contains. Adding contributions rewards candidates appearing near the top of multiple lists. The smoothing constant k limits how strongly a single first-place rank dominates. The original RRF paper provides the method; the lab implements a transparent variant with deterministic ties and access filtering.

If document A ranks first in one list and third in another, while B ranks second in both, their contributions can be calculated directly. Rank fusion loses information about score margins, so it is not automatically better than a calibrated weighted-score model. It is a robust baseline when heterogeneous score scales are difficult to align.

Run sparse and dense retrieval under the same access and time filters, merge candidates by stable chunk identity, and inspect where each method contributes unique relevant evidence. Deduplicate repeated chunks before final context selection. If one retriever returns many overlapping versions of the same paragraph, agreement may reflect indexing duplication rather than independent evidence. A fusion score should not be described as source consensus.

Evaluate sparse-only, dense-only, and hybrid retrieval on the same questions and corpus snapshot. Report per-query wins and failures as well as an average. A hybrid system adds indexing and query cost, so it should solve a concrete recall or precision gap. When it fails, inspect candidate presence first. No reranker can promote an answer-bearing passage that neither first-stage retriever included in its candidate pool.

This executable BM25 variant uses positive IDF, whitespace-like regex tokenization, and a tiny in-memory corpus. It demonstrates sparse scoring exactly; it is not dense retrieval or a scalable search engine. Change document lengths to inspect normalization.