Agentic Engineering Lab

Build it in stages

Run the synthetic comparison seed and verify every per-case result.
Write a one-page experiment card and a related-work matrix using primary sources.
Replace or extend the fixtures with an authorized task collection split by independent groups.
Add a strong simple baseline, one mechanism ablation, and a fixed resource budget.
Export per-case results, manifest, uncertainty analysis, and a report with failures and limitations.
Test the quick-start procedure from a clean environment and distinguish smoke-test results from full experiments.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

The hypothesis, primary metric, guardrails, and falsifier are written before final evaluation.
Baseline and candidate use identical case IDs with no silently dropped failures.
The manifest records code, data, model, prompt, and configuration identifiers or explicitly unavailable fields.
Reported metrics can be recomputed from per-case outputs.
The conclusion labels synthetic data and observed limitations, and makes no unsupported novelty claim.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py

python

import hashlib
import json
from itertools import product

CASES = [("a", 0, 0), ("b", 1, 0), ("c", 2, 1),
         ("d", 3, 1), ("e", 4, 1), ("f", 5, 1)]
CONFIG = {"baseline_threshold": 3, "candidate_threshold": 2,
          "fixture": "invented scalar classification", "higher_score_is_better": True}

def evaluate(threshold):
    return {case_id: int(int(value >= threshold) == label)
            for case_id, value, label in CASES}

def compare(baseline, candidate):
    differences = [candidate[key] - baseline[key] for key in sorted(baseline)]
    mean = sum(differences) / len(differences)
    extreme = sum(abs(sum(sign * value for sign, value in zip(signs, differences))
                      / len(differences)) >= abs(mean) - 1e-12
                  for signs in product((-1, 1), repeat=len(differences)))
    return {"mean_delta": mean, "p_value": extreme / 2 ** len(differences),
            "wins": sum(value > 0 for value in differences),
            "losses": sum(value < 0 for value in differences)}

def run():
    baseline = evaluate(CONFIG["baseline_threshold"])
    candidate = evaluate(CONFIG["candidate_threshold"])
    payload = json.dumps({"config": CONFIG, "cases": CASES}, sort_keys=True)
    return {"config": CONFIG, "input_sha256": hashlib.sha256(payload.encode()).hexdigest(),
            "baseline": baseline, "candidate": candidate,
            "comparison": compare(baseline, candidate),
            "limitation": "synthetic smoke study; no population generalization claim"}

if __name__ == "__main__":
    report = run()
    assert sum(report["baseline"].values()) == 5
    assert sum(report["candidate"].values()) == 6
    assert report["comparison"]["p_value"] == 1.0
    print(json.dumps(report, indent=2, sort_keys=True))

Push it further

Run an independent reproduction with a second implementation or a different environment, then publish a discrepancy analysis that can challenge the original conclusion.