feat(FM): give registered watcher checks last-run telemetry #10

Merged

john merged 5 commits from fm/fm-check-failure-telemetry into main 2026-08-17 17:58:35 +00:00

Conversation 0 Commits 5 Files changed 10 +881 -9

john commented 2026-08-17 17:57:09 +00:00

Owner

Copy link

Why

A registered watcher check is silent by design on every sweep that produces no wake — correct for the
wake stream, but it means a check that runs and produces nothing is indistinguishable from one
failing every time.

That is not hypothetical. During the 2026-08-16 overnight incident the budget check produced no output
for roughly twelve hours after the window it was watching had reset. The post-mortem
(data/fm-afk-resume-postmortem/report.md) ruled out crash, hang, rejected trust and swallowed output,
and still could not determine whether quota-axi was unusable for that whole period or the value
genuinely never crossed the threshold — because the check had no telemetry of its own and its
|| exit 0 swallowed any tool failure silently.

What this adds

Every registered check now records a per-run marker — timestamp, result, and whether its own data
source succeeded — on every sweep.

• In the machinery (bin/fm-check-lib.sh), not in individual check scripts. Checks are
  content-hash-bound by fm-check-register.sh, so putting this in the scripts would force
  re-registration and push the burden onto every future check author.
• Zero new wakes. The whole point is recording liveness without waking firstmate; verified by
  inspection that the change adds no wake call.
• Surfaced where it will be seen — a stale marker appears in the session-start fleet digest the
  same way a stale watcher beacon already does, rather than sitting in a file nobody opens.
• Teardown removes the sidecar, so it does not outlive its task.

The part that matters most

The telemetry cannot fail silently. A failing sidecar write is detected and reported rather than
swallowed, and an unparseable or empty sidecar is flagged rather than read as healthy. Without that
this change would have rebuilt the exact blind spot it exists to close — a monitor that goes quiet and
looks fine.

Tests

272 lines of new test for a 79-line change, which is the right ratio for something that will run on
every watcher sweep indefinitely. Teardown coverage extended alongside.

## Why A registered watcher check is silent by design on every sweep that produces no wake — correct for the wake stream, but it means **a check that runs and produces nothing is indistinguishable from one failing every time**. That is not hypothetical. During the 2026-08-16 overnight incident the budget check produced no output for roughly twelve hours after the window it was watching had reset. The post-mortem (`data/fm-afk-resume-postmortem/report.md`) ruled out crash, hang, rejected trust and swallowed output, and still could not determine whether `quota-axi` was unusable for that whole period or the value genuinely never crossed the threshold — because the check had no telemetry of its own and its `|| exit 0` swallowed any tool failure silently. ## What this adds Every registered check now records a per-run marker — timestamp, result, and whether its own data source succeeded — on **every** sweep. - **In the machinery** (`bin/fm-check-lib.sh`), not in individual check scripts. Checks are content-hash-bound by `fm-check-register.sh`, so putting this in the scripts would force re-registration and push the burden onto every future check author. - **Zero new wakes.** The whole point is recording liveness without waking firstmate; verified by inspection that the change adds no wake call. - **Surfaced where it will be seen** — a stale marker appears in the session-start fleet digest the same way a stale watcher beacon already does, rather than sitting in a file nobody opens. - **Teardown removes the sidecar**, so it does not outlive its task. ## The part that matters most **The telemetry cannot fail silently.** A failing sidecar write is detected and reported rather than swallowed, and an unparseable or empty sidecar is flagged rather than read as healthy. Without that this change would have rebuilt the exact blind spot it exists to close — a monitor that goes quiet and looks fine. ## Tests 272 lines of new test for a 79-line change, which is the right ratio for something that will run on every watcher sweep indefinitely. Teardown coverage extended alongside.

john added 5 commits 2026-08-17 17:57:09 +00:00

Give registered watcher checks last-run telemetry ... 9102740925

A registered custom check (state/<id>.check.sh, bound by fm-check-register.sh)
is silent by design on every sweep that produces no actionable wake. That is
correct for the wake stream but makes a check that is failing every time
indistinguishable from a check running cleanly with nothing to report. During
a 16h overnight incident the budget check produced no output for ~12h after
the window reset, and the post-mortem could not determine whether quota-axi
was unusable that whole period or the value genuinely never crossed the
threshold, because the check has no failure telemetry of its own.

Add a cheap per-check last-run sidecar (state/<id>.check-last-run) written by
the watcher machinery on EVERY invocation regardless of output, so a silent
multi-hour outage is visible without forensic log reconstruction. The record
is one line (ts=<epoch> rc=<exit> out=<0|1>), overwritten each run, and never
adds a wake. The exit code is the data-source success proxy a check author can
report without printing.

Put the recording in the machinery (bin/fm-check-lib.sh + bin/fm-watch.sh
check sweep), not in individual check scripts, so it applies to every
registered check without per-author re-registration. Surface a stale or
erroring record in the session-start fleet digest the way a stale watcher
beacon is surfaced (fm_check_last_run_report), never as an extra wake.
Teardown removes the sidecar with the rest of the check artifacts so it
cannot linger as a perpetual stale entry.

Detect a failing sidecar write so telemetry cannot fail silently ... 6807d6805b

The sidecar write is best-effort by design (it must never disrupt the check
sweep or add a wake), so a write failure is swallowed locally. But a
persistent write failure - disk full, bad perms, a never-created sidecar -
would rebuild the exact blind spot this telemetry closes: the check runs but
no record appears, and nothing says so.

fm_check_last_run_report now also flags a registered custom check
(state/<id>.check.sh bound by fm-check-register.sh, so state/<id>.check-trust
exists) that has NO sidecar even though a sweep completed after it was
registered (the watcher's .last-check is newer than the .check-trust). That
means the machinery ran the check but the sidecar write failed, and the
absence of the sidecar - which this scan detects - is what makes the failure
visible rather than silent. A check registered after the last sweep, or a
home where no sweep has ever run, is not flagged (the watcher-beacon alarm
owns the latter).

Adds a portable mtime helper local to fm-check-lib.sh so the read-only scan
owns its own reading without depending on bin/fm-wake-lib.sh. Four new test
cases pin the after-sweep flag, the before-first-sweep skip, the no-sweep
skip, and the unregistered-check skip.

no-mistakes(review): flag unparseable check sidecars, soften digest wording, test digest 4b5f545311

no-mistakes(review): name all no-telemetry causes, require full sidecar parse, honour X-mode cadence 1c8ab27757

no-mistakes(document): document check last-run telemetry in its owner docs 787386e3fd

john merged commit 3b775a9808 into main 2026-08-17 17:58:35 +00:00

john referenced this pull request from a commit 2026-08-17 17:58:36 +00:00

Merge pull request 'feat(FM): give registered watcher checks last-run telemetry' (#10) from fm/fm-check-failure-telemetry into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

john/firstmate!10

No description provided.