Incident Rehearsal

The problem worth solving

What could make it stand out

System architecture

Data and reproducibility

Start with a synthetic benchmark

• AIOpsLab research publication
• ITBench research paper

Baselines you must beat

1. A fixed incident runbook with deterministic trigger conditions and explicit preconditions for each action.
2. A direct-action bounded agent that receives the same telemetry, action vocabulary, and total observation budget.
3. A planner that simulates only expected primary-metric improvement, without dependency invariants or compensation checks.

Measure the claim

Targets below are proposed success criteria. No results have been achieved on your behalf.

Experiments and ablations

1. Use the same candidate plans to compare primary-metric-only scoring with invariant-aware rehearsal. This isolates the action-selection mechanism from differences in language-model plan generation.
2. Inject a dependency-capacity change between rehearsal and execution. Test whether postcondition verification triggers compensation and whether the incident review records the model mismatch accurately.
3. Sweep planning horizon and rehearsal sample count under a fixed compute budget. Compare safe recovery, latency, and unnecessary abstention rather than optimizing only a single aggregate score.

Your execution plan

0 of 12 deliverables checked. Tick a deliverable after recording evidence in your project repository.

Working seed

Run this small, deterministic core first. It demonstrates the central mechanism. The complete system, experiments, and deployment are your capstone work.

"""Incident Rehearsal: simulate an action plan and compensate failed execution."""
from copy import deepcopy
import json


def error_fraction(state):
    capacity = state["replicas"] * (30 if state["version"] == "bad" else 70)
    return round(max(0, state["demand"] - capacity) / state["demand"], 3)


def apply(state, action):
    if action == "scale":
        state["replicas"] = 4
    elif action == "rollback":
        state["version"] = "stable"
    elif action == "restart_lossy":
        state["pending"] = 0
    else:
        raise ValueError("Unknown simulator action")


def violations(before, after):
    errors = []
    if after["replicas"] * 45 > after["db_budget"]:
        errors.append("db_connection_budget")
    if after["pending"] != before["pending"]:
        errors.append("pending_work_lost")
    return errors


def rehearse(state, plan):
    sandbox = deepcopy(state)
    trace = []
    for action in plan:
        before = deepcopy(sandbox)
        apply(sandbox, action)
        errors = violations(before, sandbox)
        trace.append({"action": action, "violations": errors})
        if errors:
            return {"safe": False, "trace": trace, "predicted_error": None}
    return {"safe": True, "trace": trace, "predicted_error": error_fraction(sandbox)}


def execute_with_compensation(state, plan):
    snapshot = deepcopy(state)
    for action in plan:
        before = deepcopy(state)
        apply(state, action)
        if violations(before, state):
            state.clear()
            state.update(snapshot)
            return "compensated"
    return "committed"


def main():
    state = {"replicas": 2, "version": "bad", "demand": 100,
             "pending": 8, "db_budget": 100}
    original = deepcopy(state)
    scale = rehearse(state, ["scale"])
    rollback = rehearse(state, ["rollback"])
    assert not scale["safe"] and rollback["predicted_error"] == 0
    assert state == original
    assert execute_with_compensation(state, ["rollback", "restart_lossy"]) == "compensated"
    assert state == original
    assert execute_with_compensation(state, ["rollback"]) == "committed"
    print(json.dumps({"before_error": error_fraction(original), "scale": scale,
                      "rollback": rollback, "after_error": error_fraction(state),
                      "pending_preserved": state["pending"]}, sort_keys=True, indent=2))


if __name__ == "__main__":
    main()

Failure modes to investigate

• A perfectly known world makes rehearsal unrealistically powerful. Give every method the same observations and add held-out parameter changes between prediction and execution.
• Restoring an in-memory snapshot is easier than undoing real distributed effects. Define compensation field by field and do not imply that queue losses or external side effects are reversible in production.
• Recovery metrics can reward dropping work. Verify service health together with preserved acknowledged jobs, bounded dependencies, and duplicate-completion checks.

Your demo, moment by moment

1. Start an API incident where a new deployment cuts capacity and the agent proposes both scaling and restoring the prior version.
2. Rehearse scaling: latency improves but database connections exceed the allowed budget. Show the precise violated invariant and reject the plan.
3. Rehearse and execute the deployment restore in the simulator, then verify sustained recovery and preservation of pending jobs.
4. Inject a mismatch during another plan, trigger compensation, and compare the restored state with its snapshot before opening the aggregate evaluation.

Write the resume bullet after the experiment

Replace every placeholder with your actual measurements. Keep the dataset size, baseline, and evaluation conditions available for interview questions.

A research extension

Related work to challenge your idea