The Scheduled Agent Fleet That Reported Healthy and Shipped Nothing: Why No Job Gets to Grade Its Own Success
Thirty scheduled jobs logged success for days while delivering nothing. The fix was a standing rule: a job's own status report counts for zero. The artifact it produced is the only pass, and something off the box has to read it.
What was getting in the way.
A fleet of roughly thirty scheduled agent jobs goes live, and every status readout says healthy. In reality, a restart broke credential access in the scheduled context, the jobs could not do their work, and a wrapper returned success regardless of the exit code it had just received. For days the system produced zero deliverables while every instrument said green. Self-reported success is the default in most automation, and it fails in the worst possible direction: confidently.
Context
Universal. Any organization running scheduled automations, agent fleets, cron jobs, or recurring reports has instruments that grade themselves. The pattern applies wherever "the job logged success" is currently accepted as proof that work happened.
How the work runs.
- 01
Catch the lie at the artifact, not the status column.
On go-live day the status readouts showed roughly thirty healthy jobs. The run log told the truth: the failure text sat three lines above a success-coded run line, because the handoff wrapper returned success no matter what had just failed. Zero deliverables existed for any channel.
- 02
Rule out the convenient suspect by mechanism, not timing.
Another change had landed within the hour, and blaming it was tempting. The agent separated the two by code path: that change gated inbound messages, and the broken path was outbound. Coincidence in time is not cause.
- 03
Watch the fix claim fail the next day.
After a manual refire produced one good day, the next scheduled run failed again at the same line, and nothing alarmed. The failure was only noticed when the daily retrospective was written. "Remediated" had meant "a human pushed the button once," not "the system works."
- 04
Correct its own false finding.
The report of "zero artifacts for three days" turned out to be wrong: the jobs had been writing precise one-line failure markers every day (31, then 30, then 24 of them). The sweep had only looked for success records. A loud failure that nobody reads is indistinguishable from silence.
- 05
Install the standing rule.
Every scheduled job must declare the artifact it writes on success and the marker it writes on failure, and one independent reader, off the box, checks for both on every scheduled run. Self-grading is retired fleet-wide. A job saying "I succeeded" now counts for exactly nothing.
Evidence from the workflow.
Each system has a role.
the surface under audit (where success-coded failures lived)
Scheduler + job wrappers
record of truth (the only evidence that counts as a pass)
Run logs and dated artifacts
signal (machine-readable one-line diagnoses, now with a dedicated reader)
Failure markers
action (independently confirms artifact or marker for every run)
Off-box verification reader
Why this is Operator.
Owns a workflow end to end, running the process inside the authority you set.
- 01
The verification reader is read-only and runs off the box it audits, so the system being graded cannot influence its own grade. Corrections to earlier findings stay visible in the record as corrections. Credential and infrastructure fixes remained with their human owners; the agent's scope was diagnosis, evidence, and the verification rule.
Impact / Outcomes
A multi-day outage that every status readout called healthy was found, root-caused at the artifact level, and converted into a permanent fleet-wide rule instead of a one-time fix.
The verification standard moved from "the job says it ran" to "the thing the job was supposed to produce exists, and an independent reader confirmed it."
The investigation also corrected its own first conclusion in writing: the "no artifacts" finding was a measurement error, and the record shows the correction rather than overwriting it.
Detection no longer depends on a human happening to ask. Both failure modes (a job that lies about success, and a correct diagnosis nobody reads) are now covered by the same reader.
Find where a workflow like this fits.
Start with the systems, work, constraints, and authority already present in your operation.