The alarm that cannot fire:
A scheduler stopped and nothing alarmed — because a stopped scheduler emits no errors. Every alarm counted failures; none counted silence.
CI stayed green for weeks with three dead jobs, because a trailing `|| true` converts "this broke" into "this passed".
A cron mean gap of 0.77h hid a recurring 8-13 hour nightly hole, and the thing it guarded had a 3-hour deadline. Alarm on the maximum gap, never the mean.
https://eltociear.github.io/
#devops #monitoring #opensource
A scheduler stopped and nothing alarmed — because a stopped scheduler emits no errors. Every alarm counted failures; none counted silence.
CI stayed green for weeks with three dead jobs, because a trailing `|| true` converts "this broke" into "this passed".
A cron mean gap of 0.77h hid a recurring 8-13 hour nightly hole, and the thing it guarded had a 3-hour deadline. Alarm on the maximum gap, never the mean.
https://eltociear.github.io/
#devops #monitoring #opensource