Combine candidates, heuristics, and independent checks

Sampling several candidate solutions can expose alternatives that one deterministic completion misses. Self-consistency aggregates final answers across diverse sampled reasoning paths; the original paper evaluates this strategy on particular reasoning benchmarks. It does not establish that majority agreement is always correct. A production system still needs a way to normalize equivalent answers, handle ties, and decide whether the remaining disagreement matters.

Suppose five candidates propose delivery totals of 18, 18, 18, 21, and 21. Majority voting favors 18. If all three 18s forgot the same service fee, agreement reinforces the error. Candidate diversity should include differences in decomposition, evidence, or computation, not merely wording. The vote count measures agreement among generated candidates, not a calibrated probability that the answer is true.

A verifier checks a candidate against a target property. For arithmetic, recompute the expression using trusted operations. For a plan, validate prerequisites and resource limits. For an answer grounded in documents, check whether cited passages support each material claim. For code, execute relevant tests and inspect the contract. A verifier can be deterministic or learned, but its own failure modes need evaluation.

Research on training verifiers demonstrates the value of selecting among generated solutions using a learned correctness signal in a defined setting. In an engineering workflow, use a deterministic check when one can directly establish the property. Asking the same model whether its answer looks right may repeat the same misconception. Independence concerns error mechanisms, not merely whether a second prompt was used.

A scheduling task requires assigning jobs of lengths 4, 3, and 2 across two workers, with no job split. Candidates report makespans of 4, 5, and 6. The total work is nine, so the average-load lower bound is 4.5 and any integer makespan is at least five. A candidate claiming four is impossible before the assignment is even inspected. An assignment of [4] and [3,2] achieves five and supplies constructive evidence.

The code uses a simpler invoice example. Candidate totals are compared against the authoritative quantity, unit price, and fee calculation. Majority voting chooses 18, but the deterministic check accepts 21. This does not simulate language-model sampling; it supplies fixed candidate outputs so the selection mechanism and its failure case are completely reproducible.

A heuristic estimates which unfinished candidate is worth expanding. A dependency planner might favor actions likely to remove the largest bottleneck. A retrieval agent might favor a source likely to resolve a missing entity. Tree of Thoughts explores search over intermediate candidates with evaluation and backtracking; its useful architectural distinction is that generation, evaluation, and search policy can be separate components.

Heuristic scores are not proofs. If an estimate overvalues a familiar source, search may ignore a less familiar but decisive one. Preserve a small exploration budget and inspect failures by scenario type. For optimization problems, distinguish a valid lower bound from an informal promise score: only the former can justify certain pruning arguments. Do not transfer optimality claims from a mathematical search algorithm to an unconstrained language-model evaluator.

More candidates cost latency and tokens. Define a stopping rule based on verified completion, budget exhaustion, or low expected value of another attempt. For a simple calculation with a complete deterministic check, generating ten extra prose candidates adds little. For an ambiguous evidence task, a targeted retrieval action may add more value than another answer sample using the same incomplete context.

Evaluate the policy against a fixed set of tasks with recorded cost. Report accuracy and resource use together, and retain unresolved cases rather than forcing a winner. Candidate selection should improve a measurable outcome, not merely make the transcript appear thoughtful. The portfolio project asks for rejected candidates and the reason each failed, making verification evidence visible even when the final answer is concise.

Fixed candidates deliberately share a missing-fee error. Counter finds their majority, while a deterministic calculation verifies the actual total. Replace the fixture with recorded model outputs to evaluate a real sampling policy without changing the verifier.