CI/CD, controlled rollout, and failure drills

Continuous integration should produce evidence that a particular source revision works under a specified environment. Continuous delivery carries that artifact toward deployment with the required checks and approvals. For an agent service, the artifact includes application code, dependency locks, workflow definitions, prompt or skill versions, and schema expectations. A model configuration change can alter behavior even when the container digest stays the same.

Build once from reviewed inputs and promote the resulting immutable artifact where the deployment system supports it. Record its identity in traces and job manifests. Separate tests of pure workflow logic, service integration, policy enforcement, and a fixed evaluation dataset. GitHub Actions provides a workflow syntax with jobs, dependencies, permissions, and other controls, but a workflow file is only a mechanism. The team must define which evidence constitutes an acceptable release.

During a rolling update, old and new workers may process jobs from the same queue. A schema change that the old worker cannot read can break already accepted work. Version job payloads and checkpoint schemas, and plan migrations or compatibility windows. The new version should know whether it can resume a checkpoint created by an older workflow definition.

In an original example, version one stores a single summary string while version two stores an artifact reference and content hash. Deploying version two writers before old readers understand that shape can strand jobs. An expand-and-contract migration first adds compatible readers, then introduces new writes, and only later removes the old format when no active jobs need it. The exact migration depends on the system, but the ordering principle prevents code rollout from silently invalidating durable work.

A limited rollout sends a controlled portion of eligible traffic to a new version while observing predefined metrics. Compare task success, permission denials, duplicate effects, latency, cost, and queue behavior. Avoid selecting only easy tasks for the new version and then claiming it outperformed the old one. Record how traffic was assigned and whether both groups had comparable work.

A release gate can require no known invariant violation and keep error or latency changes within a declared tolerance. Small canary samples may miss rare failures, so use both deterministic failure tests and live operational observations. The accompanying code evaluates three synthetic rollout summaries against explicit thresholds. These thresholds are teaching assumptions, not universal production recommendations. A gate should state which condition failed rather than returning an unexplained red indicator.

A drill injects a known failure into an intended test environment and checks a predicted outcome. Choose one mechanism at a time: terminate a worker after a side effect, delay a queue acknowledgement, expire a lease, return a provider rate limit, corrupt a checkpoint version, or interrupt an approval wait. Define the invariant, expected detection, recovery action, and stopping condition before running the drill.

For a duplicate-delivery drill, the invariant might be one net publication for one approved artifact. The worker should recover a receipt and finish without publishing again. For a stale-worker drill, an old fencing token should fail to update state. Measure recovery time and unresolved work, not only whether the process eventually restarted. A recovered process can still leave incorrect business state or duplicate external effects behind.

Reverting a container image may restore old code, but it does not automatically revert database migrations, sent messages, published artifacts, or payments. A rollback plan must say what happens to jobs already started by the new version and whether the old version can interpret their checkpoints. If not, pausing admission and completing or migrating in-flight jobs may be safer than an immediate blind downgrade.

Keep a concise runbook with observable symptoms, permitted actions, decision ownership, and verification steps. After a drill or incident, preserve evidence and update the specific failed assumption. The project seed demonstrates duplicate job processing against an in-memory SQLite table, which is useful for transaction reasoning but does not survive process loss. The full project adds durable storage, a real API and worker boundary, reproducible packaging, and a drill report. Production readiness is supported by that operational evidence, not by the presence of Docker and a green build badge alone.

Three invented summaries illustrate a release policy with explicit failure reasons. The thresholds are hypothetical, and the example does not perform a deployment or infer statistical confidence from a sample. A real gate also needs validated measurements, sample-size rules, and version compatibility checks.