Workspace/Lesson workspace
Loading progress
Acting & collaborating40 min

Design triggers that avoid repeated and unsupported action

Lesson 2 of 3
How to study this lesson

Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.

By the end, you can

  • Use hysteresis and cooldown for stable triggering.
  • Separate evidence thresholds from authorization policy.
  • Evaluate false alerts, missed events, and notification burden.

A trigger converts evidence into a decision point

A trigger is a rule for when the system should evaluate or propose work. It may fire after a count threshold, a state change, a period without activity, or a scheduled interval. The triggered agent then interprets evidence and prepares an appropriate response. Keeping the trigger deterministic makes it easier to explain why a run started and to test cases where no run should occur.

Suppose a service monitor sees an error rate rise above ten percent. A trigger can open an investigation using the recent window and baseline. It should not immediately assume a particular root cause or make a deployment change. The evidence threshold answers whether attention is warranted; an action policy answers what the agent may do with that attention. Those are separate questions with different inputs and different evaluation criteria.

Hysteresis prevents threshold chatter

A single threshold can oscillate when values move around its boundary. With a ten percent threshold, readings of 9.9, 10.1, 9.8, and 10.2 can repeatedly enter and leave alert state. Hysteresis uses separate enter and reset thresholds, such as alert at ten and rearm at five. After alerting, the system remains latched until the metric falls sufficiently, preventing repeated alerts for one continuing episode.

Add a cooldown to limit alert frequency after separate episodes. If an alert occurs at second ten with a sixty second cooldown, another eligible episode cannot alert before second seventy. Decide whether an episode detected during cooldown should be dropped, queued, or allowed to alert later if it still persists. The example keeps the trigger armed until it can emit, which is one explicit policy rather than a universal definition of hysteresis.

Build an evidence package for the agent

A triggered run should receive the aggregate, baseline, window boundaries, source health, and representative events. If a metric is twelve percent but half the source partitions are missing, the agent needs that caveat before interpreting the change. Include the trigger version and threshold configuration so later reviewers can reproduce why the run began. A natural-language explanation can be generated from these facts without inventing causal certainty.

Consider twenty failures out of one hundred requests versus two failures out of ten. Both rates are twenty percent, but their evidence strength and operational context differ. A minimum sample rule can prevent noisy small denominators from triggering an expensive investigation. The right minimum depends on the task and should be evaluated using real labeled histories. The synthetic numbers in this lesson illustrate design choices, not a validated operational threshold for every service.

Policy constrains the resulting action

Define action classes such as inspect, draft, notify, and modify. An installed monitor may be authorized to inspect metrics and prepare an internal draft, while sending to a person or changing infrastructure requires the relevant authorization. The policy should consider the user intent that established the monitor, destination, payload, data sensitivity, current permissions, and any required approval state. A model confidence score is not a substitute for those facts.

When a concrete proposal needs approval, show the exact action, target, and evidence. Preserve it as a versioned record so the user can approve the reviewed payload rather than an evolving plan. If the user already authorized a class of actions within clear bounds, apply that authorization instead of asking again for every routine event. Proactivity should reduce unnecessary attention costs while keeping meaningful decisions tied to explicit, reviewable intent.

Evaluate alert quality over a history

Replay a labeled event history and measure how many relevant incidents were detected, how many alerts were unnecessary, and how much time elapsed before useful detection. Also measure repeated notifications per incident and investigation cost. A trigger that catches every incident by alerting on every event is usually not useful. A very quiet trigger can look pleasant while missing the events the user installed it to catch.

Compare threshold, hysteresis, cooldown, and minimum-sample settings on the same history. Keep the agent's interpretation evaluation separate from the trigger evaluation: one can select the right incident while the other produces an unsupported explanation. Document cases where abstention or a data-quality warning is the correct result. The code below isolates the trigger state machine with a fixed trace so its behavior can be understood before any model, notification service, or external action is introduced.

Work through the code

This deterministic teaching simulation uses invented percentage readings. It rearms at or below five, alerts at or above ten, and waits out a sixty-second cooldown while remaining armed. It performs no monitoring, model calls, or notifications.

m20_lesson2.py
python
samples = [(0, 3), (10, 11), (20, 13),
           (30, 4), (40, 12), (80, 12)]
enter = 10
reset = 5
cooldown = 60
armed = True
last_alert = None
alerts = []

for timestamp, value in samples:
    if value <= reset:
        armed = True
    cooled = last_alert is None or timestamp - last_alert >= cooldown
    if value >= enter and armed and cooled:
        alerts.append((timestamp, value))
        last_alert = timestamp
        armed = False

print('alerts:', alerts)
print('policy: wait while armed during cooldown')
print('external actions:', 0)
EXPECTED / ILLUSTRATIVE OUTPUT
alerts: [(10, 11), (80, 12)]
policy: wait while armed during cooldown
external actions: 0

Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.

Pause and reason

With enter=10, reset=5, and cooldown=60, an alert fires at t=10. Values are 4 at t=30, 12 at t=40, and 12 at t=80. Under the example policy, when is the next alert?

Check your understanding

What does crossing an anomaly threshold establish?

Your notes

Explain the mechanism in your own words. Add a failure you want to test.

Saved notes appear in your notebook

Go deeper with primary sources

Practice this module