State machines make recovery a design problem
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Define legal transitions and terminal states.
- Correlate pending work with observations.
- Explain recovery and cancellation without claiming impossible guarantees.
Name states according to what is known
A run state should express a fact the system can support. Ready means no work has started. Running means the controller can accept a next proposal. Waiting means a tool call has been dispatched and an observation is pending. Completed means the completion checks passed. Failed and cancelled are distinct terminal outcomes. These names help both code and users understand what actions are currently possible.
Avoid a single boolean called done. It cannot distinguish success from failure, and it says little about work still happening after a request disconnects. A state machine defines allowed transitions explicitly. Ready can become running or cancelled. Running can become waiting, completed, failed, or cancelled. Waiting can return to running on a matching observation, or become failed or cancelled according to the operation's cancellation semantics.
An event must be valid for its current state
The same event can be valid in one state and invalid in another. A tool result is expected while waiting for that tool's call ID. It is suspicious while ready, after completion, or for a different call ID. A final answer cannot be accepted while a required observation remains pending. These constraints should be ordinary code checks, not assumptions left in comments.
Consider the trace start, call c1, result c2, finish. The result must be rejected because c2 does not match the pending c1. If the controller merely checks that some result arrived, it can attach unrelated evidence to the run. IDs are part of the state transition contract. Later systems with concurrent tools need a set or map of pending operations rather than the single pending slot used in this module.
Persist before and after important boundaries
For an external action, there is a gap between deciding to execute, actually executing, and recording the outcome. A crash can occur in any gap. Persist an operation identity and intended parameters before dispatch, then store the observed result afterward. On recovery, reconcile pending operations with the target system where possible instead of blindly assuming they failed.
An idempotency key can help a supporting target recognize retries of the same logical action, but it does not create a universal exactly-once guarantee across arbitrary systems. If the target has no idempotency or status lookup, an ambiguous write may require human reconciliation. The course's first agent uses read-only tools so you can learn lifecycle handling before adding these more demanding side-effect semantics.
Cancellation is a request with consequences
A user pressing Cancel should cause the controller to stop proposing new work and request cancellation of work it owns. Some operations can be interrupted promptly; others may already have committed an external change. A cancelled UI state must not claim that all effects were rolled back unless the system has evidence for that claim.
Separate cancellation requested from cancellation confirmed when your runtime needs that distinction. Preserve the last observation and any unresolved operation identity. A browser disconnect is also not automatically a user cancellation: the user may reconnect, or a mobile network may briefly fail. Decide whether runs are tied to the HTTP connection or survive it, then make the API and persistence model reflect that choice.
Replay turns traces into executable evidence
A deterministic state reducer applies events to a previous state and returns a new state without performing I/O. Tools and persistence live outside it. This makes event sequences easy to test, including invalid transitions that may be difficult to trigger through a live UI. The lab adds budgets, call IDs, and evidence requirements to this basic idea.
The example uses an Enum and a transition table to reject completion while waiting. It is intentionally smaller than a durable workflow engine: there is no database, lease, distributed scheduler, or retry system. Its purpose is to make legal order visible. Before adopting a framework, write down the same states and invariants so you can evaluate whether the framework's recovery behavior matches your application's needs.
Work through the code
The transition table is a pure in-memory model. It permits start, call, observation, finish, and cancellation, while rejecting finish from waiting. FAILED is listed as a terminal state for the fuller lifecycle but this tiny demo has no failure event. The lab implements additional event validation and correlation.
from enum import Enum
class State(str, Enum):
READY = "ready"
RUNNING = "running"
WAITING = "waiting"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
TRANSITIONS = {
(State.READY, "start"): State.RUNNING,
(State.RUNNING, "call"): State.WAITING,
(State.WAITING, "observe"): State.RUNNING,
(State.RUNNING, "finish"): State.COMPLETED,
}
def advance(state, event):
if state in {State.COMPLETED, State.FAILED, State.CANCELLED}:
raise ValueError("terminal state")
if event == "cancel":
return State.CANCELLED
if (state, event) not in TRANSITIONS:
raise ValueError("illegal transition")
return TRANSITIONS[state, event]
state = State.READY
for event in ("start", "call", "observe", "finish"):
state = advance(state, event)
print(event, "->", state.value)
try:
advance(State.WAITING, "finish")
except ValueError as error:
print("rejected:", error)
start -> running call -> waiting observe -> running finish -> completed rejected: illegal transition
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A browser disconnects while a tool is writing to a remote system. Why should the backend avoid automatically reporting "cancelled and rolled back"? Propose a more accurate status and recovery record.
Check your understanding
A result arrives with call_id c2 while the run is waiting for c1. What is the correct controller behavior?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.