Sagas, compensation, and the point of no return
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Model a multi-service workflow as durable local transitions.
- Distinguish compensation from database rollback.
- Design recovery for ambiguous outcomes and failed compensation.
Local success does not imply global success
An agent that books travel may reserve a room, charge a payment method, and issue a ticket through different services. Each service can commit its own transaction while another step later fails. There is no shared database transaction to roll back. A saga represents the workflow as a sequence of local actions with explicit recovery behavior. It makes partial completion visible instead of pretending that several HTTP calls form one atomic transaction.
Begin with business states: room reserved, payment captured, ticket issued, compensation pending, and needs review. Store stable operation identifiers and service receipts at every transition. The agent can propose the workflow, but ordinary code should enforce which transitions are legal. A natural language explanation of what probably happened is not an adequate source of truth for recovery.
Compensation is another forward action
Refunding a charge does not erase the original charge from history. Cancelling a reservation can have a fee, and a sent notification cannot be made unread. Compensation is a new business action that restores an acceptable condition, possibly with consequences. It is different from rolling back uncommitted database changes. Specify what acceptable means before building the orchestrator.
In a toy workflow, a room reservation reduces available rooms from five to four and a payment capture records 120 units. Ticketing fails. The compensators refund 120 and release the room, leaving five rooms and a zero net charge. The ledger should still show the capture and refund. Replacing the state with a pristine snapshot would conceal the history and could overwrite changes made by another booking while this workflow was running.
Order follows dependencies
Compensating completed steps in reverse order is a useful default when later steps depend on earlier ones. It is not a universal law. A business might need to cancel fulfillment before refunding payment or might allow independent reservations to be released concurrently. Document dependency constraints and the point after which recovery must complete forward.
Call this point the pivot for your workflow. Before it, completed work may be compensable. After it, the system might need to finish remaining actions because the main commitment is irreversible. For a digital entitlement, issuing the entitlement could be the pivot; sending a receipt afterwards is retryable delivery work. Make the pivot a business decision, not whichever function happens to appear in the middle of a Python list.
The hard case is a lost acknowledgement
Suppose capture succeeds remotely, but the worker crashes before saving its receipt. Replaying capture without duplicate protection may charge twice. Marking capture complete before sending creates the opposite problem: recovery could skip an action that never happened. Durable orchestration alone does not eliminate this gap. The external service needs an idempotency key or a reconciliation API that can resolve the ambiguous operation.
Compensators have the same failure modes. A refund can time out after committing and must be reconciled before repetition. Persist compensation status and attempt identifiers independently from forward steps. If compensation is exhausted, transition to an explicit review state with the known receipts and unresolved effects. Do not report that everything was undone merely because every compensator was invoked. Invocation and confirmed outcome are different events.
Use the simulator to expose assumptions
The example below models a reservation and capture in memory, then compensates them after ticketing fails. It is deterministic and useful for learning order and recording a journal. It is not a durable saga engine: a process crash loses all its data, concurrent writers are absent, and the fake operations have no uncertain acknowledgements. Those omissions are deliberate boundaries for the exercise.
A portfolio extension should inject a crash before and after each operation, including compensation, and compare observed state with the intended invariant. Separate user-visible outcome from internal cleanup progress. For instance, booking failed and refund pending is more accurate than booking failed and rolled back. The strongest evidence is a recovery matrix showing each crash point, what persisted, how the system reconciled it, and whether any action occurred twice. That evidence makes the design reviewable.
Work through the code
Two fake operations update an in-memory state and register compensators only after completing. Ticketing then fails, so the compensators run in reverse order. Change the failure location to study partial completion. A real system must persist transitions and reconcile ambiguous remote outcomes.
state = {"rooms": 5, "net_charge": 0}
journal = []
completed = []
def reserve():
state["rooms"] -= 1
journal.append("reserve")
def release():
state["rooms"] += 1
journal.append("release")
def capture():
state["net_charge"] += 120
journal.append("capture")
def refund():
state["net_charge"] -= 120
journal.append("refund")
try:
for action, undo in [(reserve, release), (capture, refund)]:
action()
completed.append(undo)
raise RuntimeError("ticket unavailable")
except RuntimeError:
for undo in reversed(completed):
undo()
print(journal)
print(state)
['reserve', 'capture', 'refund', 'release']
{'rooms': 5, 'net_charge': 0}
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A receipt email fails after a paid digital license has been activated. Deactivating the license would interrupt a user who already started using it. Define a sensible pivot and recovery policy.
Check your understanding
A refund request times out after a failed booking. What is the most accurate next state?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.