Scheduled Alert System — Shadow-Mode Backtest Before a Single Alert Reaches a Human
An alert system is a trust system. Prove it quiet before it ever gets loud.
What was getting in the way.
A daily-alert system for a complex paid-media account needs twenty-plus detectors across four platforms. Shipped the obvious way — threshold today, send to the channel tomorrow — it floods the channel on day one with warnings nobody can act on, and some of the detectors are measuring the wrong thing in ways nobody will notice until a decision is made against a false signal. The usual outcome: the channel mutes the alert, and real signals die with the noise.
Context
Universal — the pattern applies to any always-on alert system where false positives are permanent trust damage.
How the work runs.
- 01
Build every detector first, pin thresholds second
The agent writes the detector logic for every signal across every platform — pacing, delivery, feed health, conversion drift, benchmark gaps — before anyone argues about thresholds, so threshold tuning happens on a complete picture rather than one detector at a time.
- 02
Replay ninety days of history through the live system
Every detector runs against the real last-ninety-days data, with the thresholds everyone agreed in the morning, and the agent counts how many times each one would have fired. The alert rate target is set first (one to three alerts per week); the backtest grades the configuration against the target.
- 03
Catch detectors that are measuring the wrong thing
Any detector firing on most days of the backtest is treated as a definition problem, not a threshold problem. In practice this surfaces category errors — comparing conversion value from one platform against revenue from another as if they were the same unit — that typesetting a threshold would never catch.
- 04
Tune to the target rate before anyone sees an alert
Thresholds are adjusted and the backtest re-runs until the rate matches the target. Detectors that are measuring the wrong thing are rewritten or dropped. The configuration lands at the target rate before a single message is ever sent.
- 05
Ship in shadow mode, cut over on evidence
The scheduled job runs for a short period logging what it would have alerted, visible to the operator but silent to the channel. Only when the shadow log matches the backtest does the channel delivery switch on.
Evidence from the workflow.
Each system has a role.
Signal (detector inputs)
Ad platforms (search + marketplace + display)
Signal (benchmark anchors)
Analytics + revenue sources
Intake (the ninety-day replay)
Historical warehouse
Action (where alerts eventually land)
Scheduler + chat channel
Why this is Operator.
Owns a workflow end to end, running the process inside the authority you set.
- 01
No alert reaches a channel until the shadow-mode log matches the backtest expectation. Thresholds are set against a stated weekly rate target, not against yesterday's gut call. Category errors (unit mismatches, cross-source comparisons) are treated as definition bugs and rewritten, not papered over by moving the threshold.
Impact / Outcomes
An alert configuration that would have fired nineteen times a week against a one-to-three-per-week target lands at the target rate before anyone's phone buzzes.
Category-error detectors (one firing on eighty-five of ninety days) get rewritten or removed before they ever teach a human to distrust the system.
The channel's trust-at-first-alert is preserved — the first message a human sees is a real signal, because every false pattern was caught in replay.
Find where a workflow like this fits.
Start with the systems, work, constraints, and authority already present in your operation.