Skip to content

Lab: Putting the Healthcheck Script to Work Under Real Failure

The Healthcheck Script project built healthcheck.sh from scratch, function by function. This lab assumes that script already exists and asks a different question: does it actually behave correctly the moment something is genuinely wrong, running unattended, on a schedule, with no one watching it execute? Building a monitoring script and deploying one that can be trusted are two different skills — this lab is entirely about the second one.

Goal and End Result

By the end of this lab, healthcheck.sh will run automatically every five minutes via a systemd timer, and you will have deliberately triggered its WARNING and CRITICAL paths under three independent, realistic failure conditions — high disk usage, high memory usage, and a stopped service — confirming in each case that the script's exit code and output correctly reflect reality.

Topology and Starting State

A single lab VM with the healthcheck.sh script from Healthcheck Script already built. If you have not completed that project, build it first — this lab tests a script that already exists, it does not construct one.

mkdir -p ~/lab/healthcheck-deploy
cp ~/lab/healthcheck-project/healthcheck.sh ~/lab/healthcheck-deploy/
chmod +x ~/lab/healthcheck-deploy/healthcheck.sh

Requirements

  • the completed healthcheck.sh script;
  • a lab-only systemd service you can stop on demand (reuse greeter.service from Service Management Lab, or create a trivial one).

Security and Snapshot Note

This lab schedules a read-only script via a user-level systemd timer — no system-wide state is modified, and cleanup is limited to removing the timer and lab directory.

Step 1: Schedule It with a systemd Timer

# ~/.config/systemd/user/healthcheck.service
[Unit]
Description=Run healthcheck.sh

[Service]
Type=oneshot
ExecStart=/home/<username>/lab/healthcheck-deploy/healthcheck.sh

# ~/.config/systemd/user/healthcheck.timer
[Unit]
Description=Run healthcheck.sh every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

systemctl --user daemon-reload
systemctl --user enable --now healthcheck.timer

Verify:

systemctl --user list-timers healthcheck.timer

NEXT                        LEFT     LAST  PASSED  UNIT               ACTIVATES
Tue 2026-07-29 10:35:00 UTC 3min 12s n/a   n/a     healthcheck.timer  healthcheck.service

Step 2: Confirm the Baseline Passes Cleanly

systemctl --user start healthcheck.service
journalctl --user -u healthcheck.service -n 10

OK: disk (/) 42% used
OK: memory 38% used
OK: service 'ssh' is running
OK: service 'cron' is running
---
OVERALL STATUS: OK

systemctl --user show healthcheck.service -p ExecMainStatus

ExecMainStatus=0

ExecMainStatus=0 confirms systemd itself observed the correct exit code — the same discipline as reading $? directly, applied to a scheduled unit rather than an interactive run.

Why 0 / 1 / 2 specifically — the monitoring exit-code contract

The exit codes this script uses are not arbitrary: 0 = OK, 1 = WARNING, 2 = CRITICAL, and 3 = UNKNOWN is the Nagios plugin convention (linked in the References), and it is the de-facto standard every mainstream monitoring agent — Nagios, Icinga, Zabbix's external checks, most Prometheus textfile-exporter wrappers — understands without any custom parsing. This is the whole reason the lab verifies ExecMainStatus rather than trusting the printed OVERALL STATUS line: the human-readable text is for a person reading the journal, but the exit code is the machine-readable contract a monitoring system actually acts on. A script that prints OVERALL STATUS: CRITICAL but exits 0 will be treated as perfectly healthy by every automated consumer — a silent, dangerous failure that only checking the exit code exposes. Reserve 3/UNKNOWN for "the check itself could not run" (a missing dependency, an unreadable /proc file), which is a genuinely different state from "the check ran and the system is critical."

Step 3: Trigger the WARNING Path — Simulated High Disk Usage

Since deliberately filling the real root filesystem is unsafe (as Disk Usage Lab covers using an isolated loopback filesystem instead), simulate the condition by temporarily lowering the script's own threshold rather than actually filling the disk:

sed -i 's/DISK_WARN=75/DISK_WARN=1/' ~/lab/healthcheck-deploy/healthcheck.sh
systemctl --user start healthcheck.service
journalctl --user -u healthcheck.service -n 5

WARNING: disk (/) 42% used
...
OVERALL STATUS: WARNING

systemctl --user show healthcheck.service -p ExecMainStatus

ExecMainStatus=1

The exit code correctly reflects the WARNING level — verify this is genuinely observable by anything consuming the script's result, not just visible in the log text.

sed -i 's/DISK_WARN=1/DISK_WARN=75/' ~/lab/healthcheck-deploy/healthcheck.sh

Step 4: Trigger the CRITICAL Path — A Genuinely Stopped Service

Unlike the simulated disk condition above, this failure is triggered for real, since stopping a lab-only service has no real consequence:

sudo systemctl stop greeter.service

sed -i 's/SERVICES=("ssh" "cron")/SERVICES=("ssh" "greeter")/' ~/lab/healthcheck-deploy/healthcheck.sh
systemctl --user start healthcheck.service
journalctl --user -u healthcheck.service -n 5

OK: disk (/) 42% used
OK: memory 38% used
OK: service 'ssh' is running
CRITICAL: service 'greeter' is not running
---
OVERALL STATUS: CRITICAL

systemctl --user show healthcheck.service -p ExecMainStatus

ExecMainStatus=2

Restore the service and confirm recovery is also detected:

sudo systemctl start greeter.service
systemctl --user start healthcheck.service
journalctl --user -u healthcheck.service -n 5

OK: service 'greeter' is running
---
OVERALL STATUS: OK

Confirming the script correctly reports recovery, not only failure, matters just as much as the failure detection itself — a monitoring script stuck reporting CRITICAL after a real fix would generate false alarms indefinitely.

Practical Scenario: Wiring the Timer's Result into an Alert

Problem. Running via a timer produces a result in journalctl, but nobody is watching the journal continuously — the result needs to actively notify someone on CRITICAL, not wait to be read.

Command:

cat <<'EOF' > ~/lab/healthcheck-deploy/alert-wrapper.sh
#!/bin/bash
/home/<username>/lab/healthcheck-deploy/healthcheck.sh > /tmp/healthcheck-last.txt
status=$?
if [ "$status" -eq 2 ]; then
    logger -t healthcheck-alert "CRITICAL: $(cat /tmp/healthcheck-last.txt | tr '\n' ' ')"
fi
exit "$status"
EOF
chmod +x ~/lab/healthcheck-deploy/alert-wrapper.sh
sed -i 's#ExecStart=.*#ExecStart=/home/<username>/lab/healthcheck-deploy/alert-wrapper.sh#' ~/.config/systemd/user/healthcheck.service
systemctl --user daemon-reload

Verify by repeating Step 4's service-stop trigger and confirming a corresponding entry appears:

sudo systemctl stop greeter.service
systemctl --user start healthcheck.service
journalctl -t healthcheck-alert -n 5

Jul 29 10:40:05 host healthcheck-alert[6210]: CRITICAL: CRITICAL: service 'greeter' is not running --- OVERALL STATUS: CRITICAL

Conclusion. logger writes a tagged entry directly into the system journal — from here, a real alerting pipeline would forward anything tagged healthcheck-alert to a paging system or chat channel, exactly the kind of integration Monitoring, Logging, and Automation covers at fleet scale, applied here to a single host as the minimal working version.

sudo systemctl start greeter.service

Common Mistakes

  • Testing only the OK path and assuming WARNING/CRITICAL work "because the logic looks right." As this lab demonstrates, actually triggering each branch is the only way to confirm the exit code a downstream system will actually receive matches what the code intends.
  • Forgetting to restore a lowered threshold after testing, leaving the script permanently oversensitive. Step 3's cleanup line is easy to skip in a rush — always restore test thresholds before considering a test complete.
  • Alerting on every run instead of only on state changes. The wrapper above alerts on every single CRITICAL run, which would page someone every five minutes for the same ongoing issue — a more mature version would track the last known state and alert only on a transition, a natural extension left as an exercise.

Rollback and Cleanup

systemctl --user disable --now healthcheck.timer
rm -f ~/.config/systemd/user/healthcheck.service ~/.config/systemd/user/healthcheck.timer
systemctl --user daemon-reload
rm -rf ~/lab/healthcheck-deploy

Final Assessment Criterion

This lab is complete when all of the following hold:

  • the timer runs healthcheck.sh automatically without manual intervention, confirmed via systemctl --user list-timers;
  • all three exit codes (0, 1, 2) have been triggered deliberately and confirmed via ExecMainStatus, not just read from the script's text output;
  • a CRITICAL result produces a distinct, taggable journal entry via the alert wrapper, and recovery is correctly reflected once the underlying issue is fixed.

Exercises

  1. Modify the alert wrapper to track the previous run's status in a state file (/tmp/healthcheck-last-status) and only call logger when the status has changed from the previous run, eliminating the repeated-alert problem noted above.
  2. Add a fourth check to healthcheck.sh — load average, using uptime or /proc/loadavg — following the same check_* function pattern as the existing checks, and verify it integrates correctly into the overall status calculation.
  3. Explain, in a short paragraph, why verifying ExecMainStatus via systemctl --user show is a more rigorous test than only reading the script's printed "OVERALL STATUS" line.

References