Reading GAIA and SWE-bench without overclaiming

A benchmark name alone does not define an experiment. The selected split, task version, execution environment, tools, prompt, model configuration, time allowance, and grader all affect the result. A score is therefore a statement about a configured system on a particular task distribution. It is not a universal measure of intelligence or a guarantee of performance in a company workflow.

Start every benchmark report with a reproducibility manifest. Include dataset revision, harness version, model identifier, tool permissions, attempt budget, and how failures were counted. State whether the agent can browse, install dependencies, inspect the repository, or call other models. Two systems with different scaffolds may provide a useful end-to-end comparison, but the score difference cannot automatically be attributed to the base models alone.

The original GAIA paper proposes questions requiring combinations of reasoning, browsing, multimodal handling, and tool use. Its design is relevant to assistants because apparently simple questions can require several correct intermediate actions. That makes failures informative: the system may locate the wrong source, misread a file, perform incorrect arithmetic, or return the wrong final format.

Use an original analogous task to understand the mechanism: find a value in a supplied annual report, combine it with a date rule, and return one normalized identifier. Success requires evidence acquisition, parsing, reasoning, and formatting. Do not copy hidden answers into prompts or treat access to benchmark solutions as normal retrieval. Report the exact GAIA subset and protocol used, and avoid quoting historical model scores as if they describe current capabilities.

The original SWE-bench paper frames tasks around real repository issues and corresponding code changes. An evaluated system receives a codebase and an issue description and must produce a patch whose behavior is checked by the benchmark harness. This differs from generating an isolated function: repository navigation, environment setup, dependency understanding, and regression behavior can all matter.

A patch passing the selected tests is evidence about those checks, not a proof of universal correctness. The agent might satisfy a narrow test while breaking an untested case, or fail because its environment was misconfigured. Distinguish resolved tasks from infrastructure failures and preserve patch artifacts for inspection. Record the specific benchmark variant rather than treating all SWE-bench subsets as interchangeable. A coding portfolio should supplement public benchmark results with realistic tasks from its intended repository and explicit regression checks.

Contamination occurs when evaluation information is available through training, prompt development, retrieved solutions, or accidental harness access in ways that undermine the intended measurement. Public issues and patches can be present on the web, so a coding agent may retrieve a known solution instead of deriving a repair. Near-duplicate tasks can also make a supposedly held-out test less independent than it appears.

A time-based split helps when tasks are created after a fixed development cutoff, but it does not prove that all model training or retrieval sources are clean. Record what is known and what is unknown. Restrict access to answer keys and reference patches during evaluation. Search for exact and near-duplicate task overlap where feasible. Most importantly, avoid claiming that a benchmark is uncontaminated merely because you did not intentionally include it in a prompt.

Tool-assisted benchmarks can change when websites disappear, packages update, or external services behave differently. Freeze local fixtures where the protocol permits, record environment artifacts, and classify unavailable dependencies. Decide whether the goal is reproducibility in a fixed environment or resilience against the current live web; those are different experiments and should be labeled.

The code below constructs a small manifest and detects reused task fingerprints between development and evaluation sets. It is a limited exact-overlap check, not a contamination detector for model training. A clean result cannot rule out paraphrases, memorized solutions, or hidden overlap. Use it as one auditable control within a broader protocol. Your final report should explain the boundary of its evidence: what tasks were tested, what resources the agent used, and which unmeasured differences may limit transfer to another setting.

Whitespace and case normalization identify one exact toy overlap. The manifest makes resource choices explicit. The code does not download GAIA or SWE-bench, inspect model training data, or detect semantic paraphrases.