journalctl
Service Management and Deploying a Backend App as a Service both showed that systemctl status displays a service's last few log lines. But fully understanding a problem often requires much more than those few lines — an error from yesterday, everything logged since the last reboot, or events from a specific time window. That's the job of journalctl, systemd's centralized log-reading tool. Instead of hunting through separate log files, it lets you search everything collected by the systemd-journald service from one place, with one consistent command.
What You Will Learn
By the end of this article you will be able to:
- explain what
systemd-journaldis and how it differs from traditional plain-text log files; - apply the main filters:
journalctl -u,-f,-b,--since/--until; - filter by log severity (
priority); - switch journal storage from volatile to persistent;
- monitor and limit the disk space the journal consumes.
systemd-journald: the Centralized Log Collector
systemd-journald is the systemd component that collects log messages from every unit — services, the kernel, the boot process — and stores them in a structured, binary format. Unlike traditional plain-text files such as /var/log/syslog, the journal attaches extra metadata to every entry (which unit, which PID, which boot), which makes precise filtering afterward far easier.
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
Active: active (running) since Fri 2026-07-24 08:12:01 UTC; 4h ago
journalctl is the client command used to read what this service has collected; it writes nothing itself, only displays the existing journal based on a query.
Viewing Basic Output
With no arguments, journalctl shows every unit's entire log, in chronological order starting from the oldest entry, opened in a pager like less (quit with q). This is usually far too much information at once, so in practice it is almost always used together with a filter.
Filtering by Unit
-u (--unit) shows only entries belonging to the named unit — the most commonly used filter, since checking a specific service problem usually only requires that service's own logs.
Viewing several units together:
Following in Real Time
-f (--follow) behaves like tail -f — new entries appear on screen as they arrive, stopped with Ctrl+C. This is the most convenient way to watch a freshly started service, such as backend-demo.service from Deploying a Backend App as a Service, immediately.
Filtering by Time
journalctl --since "1 hour ago"
journalctl --since "2026-07-24 09:00:00" --until "2026-07-24 10:00:00"
journalctl --since today
--since and --until accept flexible, human-readable formats ("yesterday", "1 hour ago", or an exact date and time). When reconstructing what happened during a specific window — "what happened around 2 PM yesterday" — this is the fastest path.
Filtering by Boot
-b (--boot) shows only entries written since the current boot, excluding older entries from previous sessions. This is especially useful when diagnosing the boot-time problems covered in The Boot Process and GRUB.
Listing previous boots and picking one:
-b -1 shows the logs from the previous boot (one before the current one) — useful, for example, when a server rebooted unexpectedly and the cause needs to be found in that boot's last entries.
Filtering by Log Priority
-p (--priority) filters by the syslog standard's severity levels:
| Level | Name | Meaning |
|---|---|---|
| 0 | emerg | System is unusable |
| 1 | alert | Action must be taken immediately |
| 2 | crit | Critical condition |
| 3 | err | Ordinary error |
| 4 | warning | Warning condition |
| 5 | notice | Significant condition, but not an error |
| 6 | info | General informational message |
| 7 | debug | Detailed information for developers |
-p err shows the named level and everything more severe (lower-numbered) — meaning -p err actually covers emerg, alert, crit, and err all together.
Reading a Sample of Output
Jul 24 11:05:02 ubuntu-server systemd[1]: Started backend-demo.service - Backend demo application.
Jul 24 11:05:02 ubuntu-server python3[6301]: Listening on 127.0.0.1:8080
Jul 24 11:06:15 ubuntu-server python3[6301]: 127.0.0.1 - - [24/Jul/2026 11:06:15] "GET / HTTP/1.1" 200 -
Jul 24 11:12:40 ubuntu-server systemd[1]: backend-demo.service: Main process exited, code=killed, status=9/KILL
Jul 24 11:12:45 ubuntu-server systemd[1]: Started backend-demo.service - Backend demo application.
- the first field is date and time;
- the second is the hostname;
- the third and fourth (
python3[6301]) show which process (name and PID) wrote the entry; - the rest is the message itself.
The fourth line — code=killed, status=9/KILL — shows the process was terminated with SIGKILL, and the final line shows the service restarted automatically thanks to the Restart=on-failure policy. -n 5 shows only the last five entries, and --no-pager prints output directly to the terminal without a pager — convenient for scripts or quick checks.
Filters and Formats Beyond the Basics
A handful of options come up constantly in real diagnosis and in interviews:
| Invocation | What it does |
|---|---|
journalctl -xeu <unit> |
The single most-typed form after a failed start: -e jumps to the newest entries, -x adds explanatory "catalog" text, -u scopes to the unit |
journalctl -k |
Kernel messages only (the same data as dmesg, but timestamped and kept across boots) |
journalctl -g <regex> |
Keeps only entries whose message matches the pattern (--grep) |
journalctl -o verbose |
Shows every structured field on an entry, not just the message — the way to discover fields worth filtering on |
journalctl _COMM=sshd / _PID=1842 |
Matches on a structured field directly, independent of which unit logged the entry |
Field matches and filters combine, which is what makes the journal more powerful than grep over a flat file — for example, every error from one unit since the current boot:
Interview angle
Knowing journalctl -xeu <unit> cold is a small but real signal in an interview — it's the reflex first command after systemctl status shows a failure, and the -x catalog text often names the exact cause (a missing binary, a permission denial) in plain language. Being able to add "and journalctl -k for kernel or OOM messages" shows you know where a service that was killed from outside leaves its trace.
Controlling Journal Persistence
By default on Ubuntu Server, the journal is often stored only in /run/log/journal/ — that is, in memory, temporarily, and disappears after a reboot. To store it permanently on disk:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
The mere existence of /var/log/journal/ automatically switches journal storage to persistent (this is how the default Storage=auto setting behaves). This can also be set explicitly in /etc/systemd/journald.conf:
Info
Persistent storage matters for historical analysis and audit trails that must survive a reboot, especially on production servers. But it means using disk space, so it should be combined with the size limits covered in the next section.
Controlling Disk Usage
Limiting size:
--vacuum-size trims the journal down to the given size (removing the oldest entries first), while --vacuum-time removes everything older than the given age. A standing limit can also be set with SystemMaxUse= in /etc/systemd/journald.conf, in which case the journal never exceeds that size.
Common Mistakes
Searching for Something Specific with an Unfiltered journalctl
An unfiltered journalctl can return thousands of lines. Narrowing the scope with -u, --since/--until, or -p is recommended every time.
Assuming the Journal Is Stored Permanently, Then Looking for Logs After a Reboot
If /var/log/journal/ was never created, the journal only covers the current session — a reboot wipes out previous logs. Persistent storage must be enabled ahead of time if long-term analysis is needed.
Ignoring Disk Space Running Out
With persistent storage enabled but no size limit set, especially with a "chatty" (heavily logging) service, the journal can gradually fill up the disk. Checking periodically with --disk-usage and setting a limit with SystemMaxUse= prevents exactly this.
Exercises
- Use
journalctl -u ssh.service -n 20to read the SSH service's recent entries and explain the structure of each line (time, host, process, message). - Compare the output of
journalctl --since "10 minutes ago"andjournalctl -p warning -b, and explain the difference between the two. - In a test VM, switch the journal to persistent storage (by creating
/var/log/journal/), reboot (or simulate it), and confirm thatjournalctl --list-bootslists more than one boot. - Record the output of
journalctl --disk-usage, then run--vacuum-time=1dand observe how the result changes.
Verification criterion: given a described incident (a specific service, a specific time window, a specific severity), you should be able to write the single journalctl command that would answer it.
Summary
journalctl is the tool used to read all logs, in their structured format, collected by systemd-journald, from one place. -u filters by unit, --since/--until by time, -b by boot, and -p by severity — together, these four cover nearly any diagnostic query. The journal can default to volatile storage; persistence is enabled by creating /var/log/journal/ or setting Storage=persistent, while disk usage is controlled with --vacuum-size/--vacuum-time or SystemMaxUse=. The next article, systemd troubleshooting, folds this tool into a systematic process for diagnosing service failures.