Silent Jobs: Building a Health Check System That Produces Zero False Positives

We had 37 cron jobs. 2 had never run successfully. 4 had never run at all. 1 had been paused for a week with no record. The most frequent alert was from a health check script — it reported two failures every day. Both things it was checking had been deleted a month ago.

Not an ops incident report. This was my own system.

1. The Trigger

It started with a simple question: I was debugging a pipeline, found it had been producing stale output for days. The cron showed "active". The script existed. But the output hadn't changed in weeks.

I pulled up the full cron list: 37 scheduled jobs. The number looked okay. But digging in revealed:

• 2 jobs showing error status
• 1 job paused (no one remembered when or why)
• 4 jobs that had never run since creation
• The only daily health check was exiting with code 1 — for checking two things that no longer existed

Let me give you two concrete examples.

Job A: pipeline-incremental-daily.

Its config said script: pipeline-p3.sh incremental. Looks fine? The problem is the Hermes no_agent mode treats the entire string as a filename — spaces are part of the name. The cron runner could never find a file literally named pipeline-p3.sh incremental. So it never ran. No alert. No log. Just... nothing.

Job B: evo-health-daily. Every morning at 8:05 it runs a health check and exits 1, firing an alert to our QQ channel. The check tests two things:

1. SSH tunnel to laptop — which was never set up
2. A cc-sync directory — which was deleted a month ago during a cleanup

Two false positives. Every day. Exit code 1, every time.

And then people got used to it.

False positives are worse than no monitoring. Without monitoring, at least you know you're blind. With false positives, you learn to ignore the alerts. And when a real problem comes, you ignore it too.

2. Three Kinds of Silence

Looking back over the audit, the system's "silent failures" fell into three categories:

Type A: False Positives → Alert Fatigue

The evo-health-daily pattern. Check for something that doesn't exist. Report failure. Every day. People learn to ignore it.

Type B: Config Drift → Component removed, check remains

The CC collection system was cleaned up on July 14 — 6 scripts archived, data directory deleted. The health check kept checking the dead directory. System design and monitoring design have independent timelines. They naturally drift apart.

Type C: Silent Failure → Looks active, never succeeds

The pipeline-incremental-daily pattern. Cron shows "active". Script exists. But the cron runner can never find it. No alert, no log, no trace. It's like finding someone locked out of their house — you glance through the window and think "they're home."

All three types share the same root cause: health checks are hardcoded into a monolithic script, with no declarative relationship to the things they check.

3. Design Philosophy: The Lazy Engineer's Observability

When I sat down to rewrite this, I asked myself a few questions.

Why check something that doesn't exist? Because the script says so. But who remembers to update the health check when deleting a component?

There's no good answer to this. As long as the health check declaration and the component lifecycle are decoupled, zero false positives is mathematically impossible.

Is there a lazy solution — where health checks can never check non-existent things? Yes — each component declares its own checks. Component disappears = checks disappear.

This becomes the core design:

Not: a 600-line script checking everything
But: each component drops a JSON file in checks.d/, the framework auto-discovers

{
  "name": "core-system",
  "checks": [
    {"id": "disk_usage", "type": "disk_usage", "params": {"warn_at": 80, "crit_at": 90}},
    {"id": "memory_capacity", "type": "memory_overflow", "params": {"max_bytes": 2200}}
  ]
}

Delete the JSON file → stop checking disk and memory. Not "comment out the relevant section of a 600-line script." Gone.

Then there's a deeper philosophical question.

Traditional monitoring asks "is every component working correctly?" But you have 37 components — do you really care about each one's state? Maybe a better question is:

What's the most dangerous kind of failure? The kind that produces no alert.

4. The Dead Man's Switch

Traditional cron monitoring works like this:

Job completes → check exit code → non-zero → alert

This tells you "job failed," but never "job never ran."

Our pipeline-incremental-daily was a "never ran" victim — it never even got a chance to exit.

The Dead Man's Switch inverts the model:

Job succeeds → records a heartbeat
Monitor waits → no heartbeat within expected window → alert

This pattern is used by NASA spacecraft, nuclear plant monitoring, and many high-reliability systems. The reason: the absence of a signal is harder to detect than a bad signal.

Applied here:

That 174h gap — it was our bh-daily-health cron, paused a week ago with zero record. In "exit code" mode, pausing a job produces no error. In Dead Man's Switch mode, pause = timeout = alert.

Because you can't distinguish "intentionally stopped a service" from "forgot there was one running."

5. Architecture: Component-Declared Checks

The output is simple:

~/.hermes/health/
├── health-v2.py          # Framework (336 lines)
└── checks.d/
    ├── 00-core.json       # Disk/memory/state.db
    ├── 01-cron-heartbeat  # 10 cron heartbeats
    ├── 02-data-quality    # Pipeline/evolution log
    ├── 03-pi.json         # Pi agent freshness
    └── 04-baby-harness    # Test framework health

The framework is 336 lines of pure Python stdlib — zero dependencies. It supports 8 check types:

First run results:

V2 framework: 19 checks, 18 passing, 1 real error
False positives: 0

The single error? bh-daily-health — exposed by Dead Man's Switch as 174h overdue. A meaningful alert.

Compare with the old system-health.py:

V1: 3 errors, 1 warning
  ❌ cron error × 2 (one real + one false positive)
  ❌ memory overflow (real, fixed)
  ⚠️  cron paused (real, resumed)

V2: 1 error
  ❌ bh-daily-health 174h overdue (real, recovering)

False positive rate: 33% → 0%.

6. Why This Matters

This sounds like a small thing — just a monitoring script rewrite. But it represents a class of problems worth discussing.

AI agent systems have a peculiar property: components are created automatically, configured via conversation, and run unattended. But no one is responsible for "forgetting."

Over the past year I've spun up 50+ scripts, 37 cron jobs, a self-evolution pipeline, a cross-machine sync system — most of them via a few prompts. They're fast to create. But I don't come back to ask "is that cron still alive?" on a regular cadence.

This creates a unique kind of system entropy:

prompt → cron created → it runs → no one looks again
→ a dependency changes → it stops → no one notices
→ it starts alerting → people learn to ignore → it still alerts

The core issue: the problem isn't too few alerts — it's that the connection between alerts and system components isn't precise enough.

Component-declared checks + Dead Man's Switch addresses this from two directions:

• Component-declared: the contract is bidirectional. You create it, it tells you what it needs checked. You delete it, the checks disappear.
• Dead Man's Switch: shifts from "something bad happened" to "something expected didn't happen" — which maps more cleanly onto the nature of silent failures.

7. Pitfalls (So You Don't Have To)

Datetime timezone trap

# ❌ Explodes: datetime.now() is naive, fromisoformat returns aware
age_hours = (datetime.now() - last_dt).total_seconds() / 3600
 
# ✅ Safe: use timestamps across timezones
age_hours = (datetime.now().timestamp() - last_dt.timestamp()) / 3600

In Python ≥3.11, datetime.fromisoformat preserves timezone info. datetime.now() doesn't add any. Subtract them → TypeError. This is effectively a mandatory trap for anyone parsing ISO timestamps.

The no_agent parameter trap

The cron script field interprets spaces as part of the filename. Never write script: "pipeline-p3.sh incremental" — create a wrapper script that calls it with the argument.

The third path for false positives

I initially tried to "just fix the two false positive checks." But fixing one false positive doesn't scale — today it's laptop SSH, next month it's something else. The real problem isn't "this specific check is wrong" — it's "there's no declarative relationship between checks and components." That's why rewriting was the right call.

Summary

If you have a system monitoring script that's accumulated more than 3 checks and hasn't been significantly reworked in a couple of years — it likely has false positives.

How to tell: Is there an alert on your dashboard that you've learned to ignore?

If so, that alert is worse than no alert at all.

The framework lives at ~/.hermes/scripts/health-v2.py. The component-declared pattern is generic — swap the check definitions and it works for any project.