Resume collaboration from durable state

A coordinator can crash after a worker finishes, while an approval is pending, or during a merge. Keeping the entire run inside Python objects makes each of these events a potential restart from zero. Durable execution records enough information to reconstruct progress: task identifiers, input versions, completed outputs, retry counts, waiting conditions, and the next legal transition. Persistence is especially valuable when work spans minutes or requires a human response.

A checkpoint is a snapshot of application state, while an event log records changes from which state can be reconstructed. Either can support recovery if writes are consistent and schemas are versioned. LangGraph distinguishes thread-scoped checkpoints from cross-thread stores; in-memory variants do not survive process restart. [LangGraph persistence documentation](https://docs.langchain.com/oss/python/langgraph/persistence). The teaching example uses serialization to demonstrate a boundary, without claiming disk durability.

Consider a worker that posts a ticket and then marks its task complete. If the process dies after the ticket service accepts the request but before the completion checkpoint, a replay sees an unfinished task and may create a second ticket. Reversing the order is also incorrect: checkpointing completion before posting can lose the ticket entirely. A local checkpoint cannot atomically commit a write to an unrelated external service.

Use an operation identifier that remains stable across retries and pass it to a sink that supports idempotency, or reconcile the sink using a durable business key. A ledger records the intended operation, request fingerprint, and observed result. The same key with a different payload should be rejected because it probably indicates a bug. This gives a precise recovery strategy instead of an unjustified claim of universal exactly-once execution.

Not every step should be re-executed during recovery. A completed search should normally reuse the recorded result, including its observation timestamp. A pending read may be retried under a defined freshness policy. A random choice or model-generated plan should be stored if downstream task identifiers depend on it; generating a new plan on every restart can create a different set of effects.

Suppose run R has plan version 3 and worker W produces output hash H. Its downstream operation key could include R, the logical action name, and plan version 3. Attempt number should usually not be in that key, because changing it would defeat deduplication. If the user intentionally edits the plan to version 4, create a new authorized operation while retaining the older record. Stable identity and explicit revision are complementary, not interchangeable.

An approval pause is part of the workflow, not a sleeping process holding a browser open. Persist the exact proposed action, target, payload hash, requester, expiration rule, and current state. A later approval resumes only that proposal. If inputs or permissions changed, the executor should revalidate the action and, where necessary, prepare a revised proposal. Approval of one recipient or amount cannot silently authorize another.

The same reasoning applies to agent handoffs. A worker that returns needs-input has not failed, and a worker that finished cannot be restarted merely because an old timeout message arrives. Define allowed transitions and compare expected state versions when updating records. In distributed storage, use transactions or conditional writes to prevent two coordinators from both claiming the same pending task. A Python dictionary demonstrates the model but does not supply those concurrency guarantees.

Recovery is successful when the user can understand what happened. Expose which tasks were reused, retried, skipped, or require review. Keep run identifiers and operation identifiers in traces so an external effect can be connected to its proposal and result. Limit retention of sensitive artifacts and avoid placing credentials in checkpoints; recovery should restore workflow state, not replicate authentication secrets into every log entry.

Test crash windows with deterministic fault injection. Stop immediately before the sink, immediately after the sink, and after the completion checkpoint. The first case should perform the effect once on retry, the second should reuse or reconcile it, and the third should skip the completed task. Apply the same tests to a single-agent workflow. Durability is valuable regardless of worker count, and it may justify a simple coordinator long before it justifies a complex team of agents.

This is an in-memory teaching simulation of a sink that deduplicates operation keys. The first call simulates a crash after the effect and before checkpoint completion. JSON round-tripping models serialization only; a production sink and checkpoint require actual persistent storage.