Project: Backup Automation

The Bash Scripting module built a backup.sh script that archives a directory, verifies the archive's integrity, and cleans up old copies. At that point, the script was only ever run by hand. This module has since covered cron (cron Basics, crontab and System cron, cron Environment and Logs) and the systemd timer. It's time to combine them: turn the backup script into a fully automated process that runs unattended, on a schedule, with a result that can be verified, and a copy that reaches a second location.

Goal and End Result

By the end of this project, the following will be in place:

backup.sh, adjusted to use absolute paths so it behaves correctly under cron's environment;
a daily backup job installed through cron, running at a fixed time, with its output redirected to a file;
the same job set up as a systemd timer as well, so the two approaches can be compared side by side;
the backup archive additionally copied to a remote server with rsync;
a clear way to tell whether a scheduled run failed, and how to investigate it.

Starting State

As in the previous module's lab, work happens in dedicated test directories, which keeps real system directories out of harm's way:

mkdir -p ~/lab/backup-automation/{data,backups}
cd ~/lab/backup-automation
printf 'client list\n' > data/clients.txt
mkdir -p data/invoices
printf 'invoice-2026-07\n' > data/invoices/inv-001.txt

Copy the finished script from Project: Backup Script into this directory as backup.sh, and make it executable:

chmod +x backup.sh

Requirements

a working backup.sh script (from the earlier project), or a copy of it;
the cron service running (systemctl status cron);
a modern, systemd-based Ubuntu Server LTS system (for the timer step);
optional: a second machine, or a remote address reachable over SSH, for testing the rsync step; without one, this step can be simulated by copying to a second local directory.

Safety and Snapshot Note

Two parts of this project are destructive: cleanup_old_backups, inherited from the earlier project, which deletes old archives, and the remote rsync synchronization added here, which — if run with --delete — removes extra files on the destination side that no longer exist on the source. Both of these rules apply:

test everything against the lab directories first, never against real backups or production data;
the first run of any rsync --delete is always a --dry-run;
before installing anything as a cron job or timer, run the script by hand until it completes successfully at least once.

Step-by-Step Tasks

Step 1: Adapt the Script for cron's Environment

As covered in cron Environment and Logs, cron's PATH is shorter than a terminal's, and its working directory is different. Commands like tar and find inside the script usually aren't a problem, since they live in standard locations (/usr/bin, /bin), but the way the script itself is invoked, and the arguments passed to it, should use absolute paths only:

readlink -f backup.sh

/home/<username>/lab/backup-automation/backup.sh

Verify — run the script by hand with fully absolute paths:

/home/<username>/lab/backup-automation/backup.sh \
  /home/<username>/lab/backup-automation/data \
  /home/<username>/lab/backup-automation/backups

Expected result:

[2026-07-26 10:02:14] Backup successful: /home/<username>/lab/backup-automation/backups/data-20260726-100214.tar.gz
[2026-07-26 10:02:14] Verification OK: .../data-20260726-100214.tar.gz intact
[2026-07-26 10:02:14] Backup process complete

Step 2: Install the cron Job

crontab -e

0 2 * * * /home/<username>/lab/backup-automation/backup.sh /home/<username>/lab/backup-automation/data /home/<username>/lab/backup-automation/backups 7 >> /home/<username>/lab/backup-automation/cron-backup.log 2>&1

Three things matter in this line:

every path is absolute — the script itself, its arguments, and the log file;
>> ... 2>&1 adds a safety net on top of the script's own log() function, catching unexpected errors too (tar itself failing to start, for instance);
MAILTO is deliberately not used — output goes to a file, not email, which is the more reliable choice in production (as covered in cron Environment and Logs).

Verify:

crontab -l

0 2 * * * /home/<username>/lab/backup-automation/backup.sh ...

Rather than waiting until 2:00 a.m., temporarily change the schedule to a few minutes out to see the result right away:

crontab -e

*/2 * * * * /home/<username>/lab/backup-automation/backup.sh /home/<username>/lab/backup-automation/data /home/<username>/lab/backup-automation/backups 7 >> /home/<username>/lab/backup-automation/cron-backup.log 2>&1

After two or three minutes:

tail -n 5 ~/lab/backup-automation/cron-backup.log
journalctl -u cron.service --since "5 minutes ago" | grep backup.sh

Expected result: new backup entries in the log file, and a CMD (backup.sh ...) invocation visible in journalctl. Once confirmed, switch the schedule back to 0 2 * * *.

Step 3: Compare With a systemd Timer

Following the pattern from systemd Timers, the same job is set up as a timer:

sudo nano /etc/systemd/system/lab-backup.service

[Unit]
Description=Lab backup automation

[Service]
Type=oneshot
User=<username>
ExecStart=/home/<username>/lab/backup-automation/backup.sh /home/<username>/lab/backup-automation/data /home/<username>/lab/backup-automation/backups 7

User= matters here: without it, systemd units run as root by default, which would leave the archive files owned by root instead of the account that should own them.

sudo nano /etc/systemd/system/lab-backup.timer

[Unit]
Description=Daily schedule for lab-backup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now lab-backup.timer

Verify:

systemctl list-timers lab-backup.timer
sudo systemctl start lab-backup.service
journalctl -u lab-backup.service --since today

Expected result: list-timers shows the next scheduled run, the manually started service produces a new backup immediately, and its full output (the echo lines from the script's log() function) is visible directly in journalctl — no separate log redirection needed, unlike the cron version.

Step 4: Copy the Archive to a Remote Server

A backup that lives on only one disk provides no protection if that disk, or the whole server, fails. The archive is copied to another server (or, for testing, another local directory) with rsync, covered in scp and rsync:

mkdir -p ~/lab/backup-automation/remote-simulation

Always check with --dry-run first:

rsync -avz --dry-run ~/lab/backup-automation/backups/ ~/lab/backup-automation/remote-simulation/

sending incremental file list
data-20260726-100214.tar.gz
data-20260726-100214.tar.gz.log

sent 234 bytes  received 20 bytes  508.00 bytes/sec
total size is 457  speedup is 1.80 (DRY RUN)

If the result looks right, drop --dry-run and copy for real:

rsync -avz ~/lab/backup-automation/backups/ ~/lab/backup-automation/remote-simulation/

Against an actual remote server, the destination looks like this instead (SSH key authentication needs to be set up first, as covered in SSH Key Authentication):

rsync -avz ~/lab/backup-automation/backups/ [email protected]:/srv/backups/myapp/

This step could also be folded into backup.sh as an extra function, but it's better kept as a separate, optional step outside the script's core archiving logic — if the remote server is temporarily unreachable, the local backup should still complete regardless.

Step 5: Prove the Backup Restores, and Get Told When It Fails

Two operational realities separate a real backup system from a script that merely produces .tar.gz files.

A backup is only proven by a restore. The tar integrity check inherited from the earlier script confirms the archive isn't corrupt, but not that its contents are actually usable. Periodically restore into a throwaway directory and diff it against the source:

mkdir -p ~/lab/backup-automation/restore-test
tar -xzf ~/lab/backup-automation/backups/data-20260726-100214.tar.gz \
    -C ~/lab/backup-automation/restore-test
diff -r ~/lab/backup-automation/data ~/lab/backup-automation/restore-test/data

Empty diff output confirms the restored tree matches the original exactly. "We had backups running for a year, but none of them actually restored" is a real, career-defining failure — an untested backup is only a hope.

Silence is not success. Both the cron log file and journalctl are passive: nobody is reading them at 02:00, and a job that fails quietly looks identical to one that succeeded until the day a restore is actually needed. In production the job should actively signal failure — have the wrapper inspect the script's exit code and, on non-zero, send an alert (an email to a monitored address, a webhook to a chat channel, or an event into the monitoring system). A robust variant is a dead-man's switch: the job pings an external monitor only on success, and the monitor raises the alert if the expected ping fails to arrive on schedule — which also catches the harder case where the job never ran at all (a stopped cron daemon, a disabled timer), something a "shout on failure" approach can never detect because a job that never starts never shouts.

Troubleshooting Notes

The cron job is silent, and the log file is empty too. Check whether the job is running at all with journalctl -u cron.service --since today | grep backup.sh; if there's no entry there either, the problem is in the crontab line itself (see crontab and System cron).
The log file shows "tar: command not found" or something similar. cron's PATH is shorter (see cron Environment and Logs); use an absolute path for every external command in the script, or extend PATH at the top of the crontab.
The systemd timer version creates archives owned by root. The User= line is missing or wrong in lab-backup.service; fix it, then run sudo systemctl daemon-reload and start the service again.
rsync can't reach the remote server. First confirm a plain ssh [email protected] connection works on its own — the problem is usually the SSH key or known_hosts, not rsync itself.

Rollback and Cleanup

After testing, remove the cron and systemd timer jobs, otherwise they keep running against the test directories indefinitely:

crontab -e

Delete the line added earlier and save, then:

sudo systemctl disable --now lab-backup.timer
sudo rm /etc/systemd/system/lab-backup.service /etc/systemd/system/lab-backup.timer
sudo systemctl daemon-reload

Remove the test directories entirely:

rm -rf ~/lab/backup-automation

Final Assessment Criteria

The project is complete when:

backup.sh runs without error when invoked through cron, using only absolute paths;
the cron job's result is confirmed both in the log file and in journalctl -u cron.service;
the same job also runs as a systemd timer, and its result is visible through journalctl -u lab-backup.service with no separate log redirection needed;
the archive is verified with rsync --dry-run before being successfully copied to another directory or server;
at least one archive has been test-restored and diffed against the source to prove it is usable;
all temporary cron and systemd units are removed after testing.

Summary

This project brought together everything covered in the module — cron syntax, user and system crontabs, cron's particular environment, and the systemd timer alternative — into one practical automation chain: the script now runs unattended, on a schedule, its result can be verified, and the archive is copied somewhere else too. The next module turns to Linux security — the principle of least privilege, sudo configuration, and password policies.