Represent uncertainty before choosing an intervention
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Distinguish missing evidence, noisy evidence, and model uncertainty.
- Update a small belief distribution from an observation.
- Explain why probabilities need calibration and stated assumptions.
Uncertainty has several sources
An agent may lack a measurement, receive a noisy measurement, or use an imperfect model that maps measurements to causes. These are different problems. Missing evidence suggests gathering information. Noisy evidence may require repeated or independent measurement. Model uncertainty may require a different hypothesis class or escalation. A single free-form confidence adjective cannot tell the controller which response is appropriate.
For a small diagnostic simulation, let the hidden cause be either a deployment fault or a traffic surge. A belief distribution assigns probability to each possibility. This is a mathematical representation used by the designer; a language model writing 80% confident does not automatically create a calibrated probability. Probabilities become useful when their origin, assumptions, and behavior against observed outcomes can be inspected.
A numerical update makes assumptions visible
Suppose prior probability of deployment fault is 0.3 and traffic surge is 0.7. An observation is four times as likely under a deployment fault: likelihoods are 0.8 and 0.2. Multiply prior by likelihood to obtain unnormalized weights 0.24 and 0.14. Dividing by their sum, 0.38, yields posterior probabilities about 0.632 and 0.368.
The observation changes the preferred explanation, but it does not prove deployment fault. The likelihood values are assumptions in this classroom example. In a real diagnostic model they would need estimation, validation, and monitoring. A precise calculation on invented probabilities remains a precise calculation on assumptions. Report both the mechanism and where its inputs came from so users do not mistake numerical detail for established evidence.
Repeated evidence may not be independent
If two dashboards display the same underlying metric stream, they are not two independent observations merely because their URLs differ. Multiplying likelihoods as though they were independent can create unjustified certainty. Preserve provenance through transformations so the controller knows which observations share an origin. A summary, a cached chart, and a copied alert may all trace to one measurement.
Consider the posterior above. Reusing the identical alert as another independent observation would further increase the deployment probability without adding information. The correct response is to recognize the duplicate or model dependence explicitly. This is one reason memory retrieval and evidence handling belong in the architecture: a probability calculator cannot detect that two apparently different text snippets came from the same event unless the data model preserves that relationship.
Calibration concerns groups of decisions
Calibration asks whether predictions assigned a probability near a value occur at about that frequency across comparable cases. It cannot be established from one convincing example. Collect held-out predictions and outcomes, group them carefully, and compare predicted confidence with observed frequency. Also inspect discrimination: a model can be calibrated yet unhelpful if it assigns the same base rate to every incident.
Split evaluation by service, incident type, and time when those differences affect the model. A probability mapping fitted on quiet periods may behave differently after a product launch. Preserve an abstain or gather-more-information option when evidence is weak or outside the evaluated setting. The best next action may be a cheap discriminating read even when one hypothesis currently has the largest probability.
Beliefs support decisions, not just labels
A belief state becomes operational when combined with action consequences. Rolling back a deployment might help a deployment fault and harm a traffic surge by reducing capacity. A read-only configuration inspection costs time but can distinguish causes. The decision component should consider both uncertainty and the cost of being wrong. Selecting the most likely label and immediately acting on it discards this structure.
The code performs one finite Bayesian update and prints the posterior. It rejects an impossible all-zero evidence model because normalization would be undefined. It does not estimate likelihoods or solve a partially observable planning problem. Extend it with a second genuinely independent observation, then compare that result with accidentally counting the same source twice. The difference reveals how evidence provenance can materially alter an apparently simple numerical decision.
Work through the code
The input probabilities and likelihoods are synthetic, known numbers. Multiplication and normalization produce a posterior over two causes. This demonstrates the update equation only; it neither calibrates a language model nor validates the assumed diagnostic likelihoods.
prior = {"deployment": 0.3, "traffic": 0.7}
likelihood = {"deployment": 0.8, "traffic": 0.2}
def update(prior, likelihood):
if set(prior) != set(likelihood):
raise ValueError("states must match")
weights = {}
for state in prior:
if prior[state] < 0 or likelihood[state] < 0:
raise ValueError("negative weight")
weights[state] = prior[state] * likelihood[state]
total = sum(weights.values())
if total <= 0:
raise ValueError("observation impossible under model")
return {state: weight / total for state, weight in weights.items()}
posterior = update(prior, likelihood)
for state, probability in posterior.items():
print(state, f"{probability:.3f}")
deployment 0.632 traffic 0.368
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
The same monitoring event appears in a dashboard, a ticket, and a memory summary. Why should an uncertainty model not count all three as independent confirmation?
Check your understanding
A model says it is 90% confident in one diagnosis. What establishes calibration?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.