Skill Regression Foundry

The problem worth solving

What could make it stand out

System architecture

Data and reproducibility

Start with a synthetic benchmark

• Agent Skills specification
• SkillsBench research paper

Baselines you must beat

1. Format-only linting of package metadata and referenced files, with no behavioral execution.
2. An aggregate-success gate that accepts any candidate whose overall pass rate matches or exceeds its parent, even if a previously passing critical case fails.
3. Paired behavioral regression tests without capability tracing, showing which privilege expansions remain invisible to final-output checking.

Measure the claim

Targets below are proposed success criteria. No results have been achieved on your behalf.

Experiments and ablations

1. Create a candidate that improves common cases while breaking a rare case-sensitive identifier. Compare the aggregate gate with a paired critical-case gate and show exactly why their release decisions differ.
2. Keep the final output correct while the candidate attempts an unnecessary shell or network capability. Test whether runtime enforcement and trace-aware review expose the violation despite a passing output verifier.
3. Repeat unchanged and neutrally rewritten skills under fixed configurations to estimate evaluation noise. Use these measurements to set an insufficient-evidence outcome instead of calling every one-run difference a regression.

Your execution plan

0 of 12 deliverables checked. Tick a deliverable after recording evidence in your project repository.

Working seed

Run this small, deterministic core first. It demonstrates the central mechanism. The complete system, experiments, and deployment are your capstone work.

"""Skill Regression Foundry: deterministic release gate over simulated traces."""
from dataclasses import dataclass
import hashlib
import json


@dataclass(frozen=True)
class Version:
    name: str
    markdown: str
    transform: object
    simulated_tool_trace: tuple


def preserve_identifier(row):
    return {"id": row["id"].strip(), "tag": row["tag"].strip().casefold()}


def destructive_normalization(row):
    return {"id": row["id"].strip().upper(), "tag": row["tag"].strip().casefold()}


CASES = [
    ({"id": " Ab7 ", "tag": " Blue "}, {"id": "Ab7", "tag": "blue"}),
    ({"id": "a-2", "tag": "RED"}, {"id": "a-2", "tag": "red"}),
    ({"id": "001", "tag": " GREEN "}, {"id": "001", "tag": "green"}),
]


def evaluate(version):
    passed = [version.transform(raw) == expected for raw, expected in CASES]
    # Permissions are an external harness policy, not self-granted skill prose.
    unauthorized = sorted(set(version.simulated_tool_trace) - {"read_fixture"})
    return {"version": version.name, "sha256": hashlib.sha256(version.markdown.encode()).hexdigest(),
            "passed": sum(passed), "total": len(passed), "case_results": passed,
            "unauthorized_tools": unauthorized}


def release_gate(baseline, candidate):
    before, after = evaluate(baseline), evaluate(candidate)
    regressions = [i for i, (old, new) in enumerate(zip(before["case_results"], after["case_results"]))
                   if old and not new]
    accepted = not regressions and not after["unauthorized_tools"]
    return {"accepted": accepted, "regressed_case_ids": regressions, "candidate": after}


def main():
    # These strings are inert fixtures; no skill is installed or executed.
    header = "---\nname: normalize-fixture\ndescription: Clean a synthetic table.\n---\n"
    stable = Version("v1", header + "Preserve identifier case; normalize tags.\n",
                     preserve_identifier, ("read_fixture",))
    broken = Version("v2", header + "Uppercase every identifier; use shell if convenient.\n",
                     destructive_normalization, ("read_fixture", "shell"))
    equivalent = Version("v1.1", header + "Trim whitespace; preserve IDs; casefold tags.\n",
                         preserve_identifier, ("read_fixture",))
    rejected = release_gate(stable, broken)
    accepted = release_gate(stable, equivalent)
    assert not rejected["accepted"]
    assert rejected["regressed_case_ids"] == [0, 1]
    assert rejected["candidate"]["unauthorized_tools"] == ["shell"]
    assert accepted["accepted"]
    print(json.dumps({"rejected_release": rejected, "accepted_release": accepted}, sort_keys=True, indent=2))


if __name__ == "__main__":
    main()

Failure modes to investigate

• Tests can mirror the skill wording and reward superficial compliance. Define independent outcome properties and hidden cases before candidate authoring, and inspect reference files for embedded answers.
• Language-model variance can create spurious regressions. Measure unchanged-version noise, repeat close comparisons, and keep an explicit insufficient-evidence release status.
• A permission list inside Markdown is not a security boundary. Enforce capabilities in the runner and report both attempted and completed operations; trace sampling alone cannot guarantee unobserved behavior.

Your demo, moment by moment

1. Open two versions of a synthetic data-cleaning skill and show the small text diff that appears harmless.
2. Run paired cases and reveal that the candidate uppercases a case-sensitive identifier, despite improving normalization elsewhere.
3. Show another candidate producing the correct output while requesting a forbidden capability; the runtime blocks it and the release gate records the attempt.
4. Accept an equivalent rewrite after the declared evidence threshold is met, then inspect hashes, traces, false-block rates, and the scope of the benchmark.

Write the resume bullet after the experiment

Replace every placeholder with your actual measurements. Keep the dataset size, baseline, and evaluation conditions available for interview questions.

A research extension

Related work to challenge your idea