Circuit breakers and strict output recovery
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Trace closed, open, and half-open breaker states.
- Validate syntax, shape, and domain meaning independently.
- Limit JSON repair to explicit, reviewable transformations.
Different failures need different controls
A retry policy helps an individual operation recover from a temporary failure. A circuit breaker changes how a caller treats a dependency after observing a pattern of failures. Imagine a document extraction service used by 20 workers. Each worker retrying responsibly can still flood that service during a prolonged outage. A breaker can reject new calls quickly, allowing the application to queue work, serve clearly marked cached data, or report temporary unavailability.
A closed breaker allows requests while collecting evidence. An open breaker suppresses requests for a recovery interval. A half-open state allows a limited probe before normal traffic resumes. These are states in a controller, not proof of the dependency's actual health. Scope the breaker carefully: one tenant's invalid documents should not make extraction unavailable for all other tenants.
Trace the recovery probe
Take a toy breaker that opens after two consecutive transient failures and waits five seconds before one probe. A failure at time zero leaves it closed with count one. A second failure at time one opens it until time six. A call at time three is rejected locally and should not count as a new downstream failure. At time six, one probe succeeds and closes the breaker. If every waiting worker becomes a probe, the recovery mechanism recreates the original burst.
A production breaker therefore needs concurrency control and an explicit failure definition. Count overloads and connection failures according to the dependency contract; do not automatically count every caller validation error. A sliding error rate needs a minimum sample size to avoid opening after one unlucky request. Partitioning too broadly causes unnecessary outages, while partitioning per request prevents the controller from learning anything.
JSON is only the first gate
Model output recovery is a separate problem. Parsing establishes whether bytes represent a supported data structure. Shape validation establishes that required fields, allowed keys, and types match the tool contract. Domain validation establishes whether the proposed action makes sense. An object with amount equal to negative ten can pass JSON parsing and an integer type check while violating a purchase contract.
Python's JSON decoder also has behaviors worth understanding: applications that require strict interoperable JSON may need to reject nonfinite constants and duplicate object keys explicitly. A boolean is a subclass of integer in Python, so an amount check using only isinstance(value, int) admits True. The example uses exact integer type checking. These small boundary decisions become consequential when generated output feeds an external mutation.
Repair must not invent intent
A safe repair policy names the transformations it permits. Removing a known outer Markdown fence or surrounding whitespace can preserve an otherwise unambiguous payload. Replacing every single quote, inserting an absent currency, or extracting the first brace-delimited substring can change meaning. A repair routine cannot know whether a missing amount should be zero or whether two conflicting amounts represent a correction.
Consider an output containing amount 40 and currency USD plus an unknown field destination. Silently discarding destination could hide an attempted change in where money goes. Reject unknown fields where the contract requires exact shape. If regeneration is allowed, send a short structured validation error and the original task constraints, give it a separate small budget, and validate again. Never treat successful regeneration as authorization for an action that was not previously approved.
Compose gates without hiding evidence
A useful pipeline records transport outcome, raw output reference, normalization applied, parsing result, schema result, domain result, and final action decision. It can then answer whether the system failed to reach the provider or reached it and received an unusable answer. That distinction guides incident response: a circuit breaker cannot repair a bad schema, and repeated JSON regeneration cannot restore a disconnected service.
The accompanying example implements a narrow parser for synthetic invoice data. It deliberately has no model calls and no external actions. A real deployment should cap payload size and nesting before costly processing, use a maintained schema validator where appropriate, and preserve enough evidence for debugging under its data retention policy. Test malformed JSON, duplicated keys, booleans, unknown fields, and boundary amounts separately. Passing one clean example says little about how the boundary behaves under failures.
Work through the code
The parser accepts an exact two-field object and optionally removes one complete outer JSON fence. It rejects duplicated keys, nonfinite constants, booleans as amounts, and unsupported currency values. This is a small domain validator, not a general JSON repair engine or financial service.
import json
def unique_object(pairs):
result = {}
for key, value in pairs:
if key in result:
raise ValueError("duplicate key")
result[key] = value
return result
def reject_constant(value):
raise ValueError("nonfinite number")
def invoice(text):
fence = chr(96) * 3
text = text.strip()
if text.startswith(fence + "json\n") and text.endswith(fence):
text = text[len(fence) + 5:-len(fence)].strip()
data = json.loads(text, object_pairs_hook=unique_object,
parse_constant=reject_constant)
if type(data) is not dict or set(data) != {"amount", "currency"}:
raise ValueError("wrong fields")
if type(data["amount"]) is not int or data["amount"] <= 0:
raise ValueError("invalid amount")
if data["currency"] not in ("USD", "EUR"):
raise ValueError("unsupported currency")
return data
print(invoice('{"amount": 40, "currency": "USD"}'))
try:
invoice('{"amount": true, "currency": "USD"}')
except ValueError as error:
print(str(error))
{'amount': 40, 'currency': 'USD'}
invalid amount
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A breaker opens after ten requests, eight of which failed because the caller omitted a required field. The team proposes lowering the threshold. Diagnose the mistake and propose a better measurement.
Check your understanding
Which proposed JSON repair is most defensible for an exact purchase contract?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.