Agentic Engineering Lab

Build it in stages

Run the seed and recompute one threshold by hand.
Separate fixture rows into training context, validation, and untouched test records with stable IDs.
Use the lab metrics to compare thresholds and include flagged counts and an explicit error-cost policy.
Choose the threshold using validation only and record the tie-breaking rule.
Evaluate the frozen threshold on test records and inspect failures by category.
Export a JSON report with dataset fingerprints and a concise explanation of limits.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

The report lists TP, FP, FN, TN, precision, recall, F1, log loss, and flagged volume.
Threshold selection never reads test labels.
A tie produces the same chosen threshold on repeated runs.
At least one fixture demonstrates why accuracy or recall alone can mislead.
Every reported rate includes its denominator or underlying counts.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py

python

import json

VALIDATION = [("v1",1,0.9),("v2",0,0.6),("v3",1,0.4),("v4",0,0.1)]
TEST = [("t1",1,0.8),("t2",0,0.2),("t3",1,0.35),("t4",0,0.45)]

def score(rows, threshold):
    counts = dict(tp=0, fp=0, fn=0, tn=0)
    errors = []
    for item_id, label, probability in rows:
        predicted = int(probability >= threshold)
        key = "tp" if label and predicted else "fn" if label else "fp" if predicted else "tn"
        counts[key] += 1
        if predicted != label:
            errors.append({"id": item_id, "kind": key})
    tp, fp, fn = counts["tp"], counts["fp"], counts["fn"]
    return {"threshold": threshold, **counts,
            "precision": tp / (tp + fp) if tp + fp else 0,
            "recall": tp / (tp + fn) if tp + fn else 0,
            "flagged": tp + fp, "total": len(rows),
            "cost": fp + 5 * fn, "errors": errors}

def build_report():
    candidates = [score(VALIDATION, t) for t in (0.3, 0.5, 0.7)]
    chosen = min(candidates, key=lambda row: (row["cost"], row["flagged"], -row["threshold"]))
    frozen_threshold = chosen["threshold"]
    test_result = score(TEST, frozen_threshold)
    return {"selection_split": "validation", "false_negative_cost": 5,
            "tie_break": "lower flagged volume, then higher threshold",
            "validation": candidates, "chosen_threshold": frozen_threshold,
            "test": test_result, "note": "Tiny teaching fixtures, not a production estimate"}

if __name__ == "__main__":
    report = build_report()
    print(json.dumps(report, indent=2))

Push it further

Add bootstrap intervals for paired changes between two score sets, preserving the same sampled example IDs for both versions.