Project: Healthcheck Script

The Backup Script project automated protecting files. This project turns attention elsewhere: is the server still healthy — is there disk space, are the essential services running, is system load normal? This kind of check doesn't need to be done by hand, several times a day — it can be handed off to a script named healthcheck.sh, which reports the result with a clear exit code a monitoring system can understand.

Goal and End Result

The script checks disk usage, memory, and the state of specified systemd services, printing each check's result separately, and finally reporting the overall state with a three-level exit code: 0 (OK), 1 (WARNING), 2 (CRITICAL). This is a convention widely used in monitoring systems like Nagios/Icinga, which makes the script easy to plug into existing monitoring infrastructure.

Starting State

mkdir -p ~/lab/healthcheck-project
cd ~/lab/healthcheck-project

This project reads real system data (disk, memory, systemd services) — no separate test environment is needed, but results will vary depending on your current system.

Requirements

df, free, systemctl — available by default on Ubuntu Server LTS;
at least one running systemd service (for example, ssh or cron) to check as a sample;
optional: running this script regularly via a systemd timer or cron.

Safety Note

This script only performs read operations (df, free, systemctl status) — it changes or deletes no files, so no special rollback measures are needed. The one thing to watch is using systemctl calls without sudo, in read-only mode only (systemctl status/is-active, not restart or stop).

Building It Step by Step

Step 1: the disk check

#!/bin/bash
set -uo pipefail

# healthcheck.sh — checks disk, memory, and service state
# Exit code: 0 = OK, 1 = WARNING, 2 = CRITICAL

DISK_WARN=75
DISK_CRIT=90
MEM_WARN=80
MEM_CRIT=95
SERVICES=("ssh" "cron")

overall_status=0   # 0=OK, 1=WARNING, 2=CRITICAL

raise_status() {
    local new_status="$1"
    if [ "$new_status" -gt "$overall_status" ]; then
        overall_status="$new_status"
    fi
}

check_disk() {
    local mount="$1"
    local used_percent
    used_percent=$(df --output=pcent "$mount" | tail -n 1 | tr -d '% ')

    if [ "$used_percent" -ge "$DISK_CRIT" ]; then
        echo "CRITICAL: disk ($mount) ${used_percent}% used"
        raise_status 2
    elif [ "$used_percent" -ge "$DISK_WARN" ]; then
        echo "WARNING: disk ($mount) ${used_percent}% used"
        raise_status 1
    else
        echo "OK: disk ($mount) ${used_percent}% used"
    fi
}

check_disk "/"

Note

set -uo pipefail is used here without set -e. The reason — commands like systemctl is-active inside this script are expected to sometimes end with a non-zero exit code (to detect a service that isn't running), and set -e would stop the entire script at exactly this "expected failure." This is a practical example of the warning from Exit Codes: set -e isn't always the right choice, especially for a monitoring task where the script itself needs to actively check for failure conditions.

Verification:

chmod +x healthcheck.sh
./healthcheck.sh

Expected result (percentages vary by system):

OK: disk (/) 42% used

Step 2: the memory check

check_memory() {
    local mem_used_percent
    mem_used_percent=$(free | awk '/Mem:/ {printf "%.0f", $3/$2 * 100}')

    if [ "$mem_used_percent" -ge "$MEM_CRIT" ]; then
        echo "CRITICAL: memory ${mem_used_percent}% used"
        raise_status 2
    elif [ "$mem_used_percent" -ge "$MEM_WARN" ]; then
        echo "WARNING: memory ${mem_used_percent}% used"
        raise_status 1
    else
        echo "OK: memory ${mem_used_percent}% used"
    fi
}

free's second line (Mem:) gives total ($2) and used ($3) memory as columns; awk calculates the percentage from the two. Call it:

check_disk "/"
check_memory

Verification and expected result:

OK: disk (/) 42% used
OK: memory 61% used

Step 3: checking service state

check_service() {
    local service="$1"

    if systemctl is-active --quiet "$service"; then
        echo "OK: service '$service' is running"
    else
        echo "CRITICAL: service '$service' is not running"
        raise_status 2
    fi
}

Checking every required service from a list (the SERVICES array):

for service in "${SERVICES[@]}"; do
    check_service "$service"
done

"${SERVICES[@]}" gives every array element as a separate word, just like "$@" for script arguments.

Verification:

./healthcheck.sh

Expected result:

OK: disk (/) 42% used
OK: memory 61% used
OK: service 'ssh' is running
OK: service 'cron' is running

Try adding a deliberately nonexistent service name to the list:

SERVICES=("ssh" "nonexistent-service")

OK: service 'ssh' is running
CRITICAL: service 'nonexistent-service' is not running

Step 4: returning the overall state as an exit code

echo "---"
case "$overall_status" in
    0) echo "OVERALL STATUS: OK" ;;
    1) echo "OVERALL STATUS: WARNING" ;;
    2) echo "OVERALL STATUS: CRITICAL" ;;
esac

exit "$overall_status"

Verification:

./healthcheck.sh
echo "Script exit code: $?"

Expected result (when every check is OK):

...
OVERALL STATUS: OK
Script exit code: 0

Practical Scenario: Connecting to a Monitoring System

Problem: the healthcheck result shouldn't be read by hand every time — it needs to feed automatically into a monitoring system (or a simple alert script).

if ./healthcheck.sh > /tmp/healthcheck-output.txt; then
    echo "Server is healthy"
else
    status=$?
    echo "A problem was detected (level: $status):"
    cat /tmp/healthcheck-output.txt
fi

The calling script doesn't need to know healthcheck.sh's internal logic — it only reads the exit code (0, 1, or 2) and the output text. This is the "return a result via exit code, details via text" principle from Functions, applied at the level of an entire script.

The Complete Script

#!/bin/bash
set -uo pipefail

# healthcheck.sh — checks disk, memory, and service state
# Exit code: 0 = OK, 1 = WARNING, 2 = CRITICAL

DISK_WARN=75
DISK_CRIT=90
MEM_WARN=80
MEM_CRIT=95
SERVICES=("ssh" "cron")

overall_status=0

raise_status() {
    local new_status="$1"
    if [ "$new_status" -gt "$overall_status" ]; then
        overall_status="$new_status"
    fi
}

check_disk() {
    local mount="$1"
    local used_percent
    used_percent=$(df --output=pcent "$mount" | tail -n 1 | tr -d '% ')

    if [ "$used_percent" -ge "$DISK_CRIT" ]; then
        echo "CRITICAL: disk ($mount) ${used_percent}% used"
        raise_status 2
    elif [ "$used_percent" -ge "$DISK_WARN" ]; then
        echo "WARNING: disk ($mount) ${used_percent}% used"
        raise_status 1
    else
        echo "OK: disk ($mount) ${used_percent}% used"
    fi
}

check_memory() {
    local mem_used_percent
    mem_used_percent=$(free | awk '/Mem:/ {printf "%.0f", $3/$2 * 100}')

    if [ "$mem_used_percent" -ge "$MEM_CRIT" ]; then
        echo "CRITICAL: memory ${mem_used_percent}% used"
        raise_status 2
    elif [ "$mem_used_percent" -ge "$MEM_WARN" ]; then
        echo "WARNING: memory ${mem_used_percent}% used"
        raise_status 1
    else
        echo "OK: memory ${mem_used_percent}% used"
    fi
}

check_service() {
    local service="$1"

    if systemctl is-active --quiet "$service"; then
        echo "OK: service '$service' is running"
    else
        echo "CRITICAL: service '$service' is not running"
        raise_status 2
    fi
}

check_disk "/"
check_memory

for service in "${SERVICES[@]}"; do
    check_service "$service"
done

echo "---"
case "$overall_status" in
    0) echo "OVERALL STATUS: OK" ;;
    1) echo "OVERALL STATUS: WARNING" ;;
    2) echo "OVERALL STATUS: CRITICAL" ;;
esac

exit "$overall_status"

Troubleshooting Tips

df --output=pcent doesn't work — this flag requires a sufficiently recent version of coreutils; on a much older system, df -h / | awk 'NR==2 {print $5}' | tr -d '%' is an alternative.
check_service always reports CRITICAL — check that the service name is written correctly (with systemctl list-units --type=service); some services work both with and without the .service suffix, but the name must match exactly.
The script stops unexpectedly with set -e — as explained above, set -e is deliberately not used in this project; adding it can cause commands like systemctl is-active, where "failure is normal," to stop the script.
Different monitoring systems may interpret the exit code differently — before connecting the script to a real monitoring system, always check its documentation for which exit code maps to which severity level.

Rollback and Cleanup

Since the script changes no system state, no rollback is needed. Simply remove the lab files:

rm -rf ~/lab/healthcheck-project

Final Assessment Criterion

The project is considered successful if all of the following hold:

disk, memory, and at least two services are checked;
each check's result is printed separately, with a clear message;
the overall state is computed from the worst (highest) level, not an average of the individual checks;
the script's exit code (0/1/2) matches the "OVERALL STATUS" shown in its output;
when a deliberately failing service is added, the script correctly identifies it as CRITICAL.

Summary

This project extended the "a distinct code for every situation" principle from Exit Codes into a three-level monitoring convention (OK/WARNING/CRITICAL), organizing several independent checks (check_disk, check_memory, check_service) cleanly through functions. The next project, Log Cleanup Script, shifts focus from checking back to cleaning up — safely removing old logs that are occupying disk space.

Project: Healthcheck Script

Goal and End Result

Starting State

Requirements

Safety Note

Building It Step by Step

Step 1: the disk check

Step 2: the memory check

Step 3: checking service state

Step 4: returning the overall state as an exit code

Practical Scenario: Connecting to a Monitoring System

The Complete Script

Troubleshooting Tips

Rollback and Cleanup

Final Assessment Criterion

Summary

References