Linux Logs: Architecture and Where to Look

When a server misbehaves, the first useful question is rarely "why did this happen" — it is "where do I even start looking." Almost every answer leads to the same place: the logs. A log is a running record of events that a system or program writes as it operates — who logged in and when, which service started or crashed, which request failed and why. journalctl already covered systemd's centralized, live log tool in detail. This article steps back and lays out the full picture: how many independent logging layers Linux actually has, where each one stores its data, and how they relate to each other — the map the rest of this module builds on.

Why There Is No Single Log

A newcomer often expects "the log" to be one obvious place. In practice, Linux logging is a stack of independent layers, each operating at a different level of the system:

These layers do not replace each other; each answers a different question. A kernel-level failure, such as a failing disk controller, will never show up in an application's log, and a bug in an application's business logic has nothing to do with the kernel at all.

What's Actually in /var/log/

ls -lh /var/log/ | head -n 15

-rw-r----- 1 syslog adm   812K Jul 24 11:02 auth.log
-rw-r----- 1 root   adm    45K Jul 24 08:12 boot.log
-rw-r----- 1 root   root   1.9M Jul 24 08:12 dmesg
-rw-r--r-- 1 root   root    24K Jul 20 06:25 dpkg.log
drwxr-xr-x 2 root   root   4.0K Jul 24 08:12 journal
-rw-r----- 1 syslog adm   2.1M Jul 24 11:05 kern.log
-rw-r----- 1 syslog adm   3.4M Jul 24 11:05 syslog
-rw-rw-r-- 1 root   utmp   0    Jul 24 08:12 wtmp

The most frequently used files:

On the RHEL/CentOS/Rocky family, file names differ slightly — secure instead of auth.log, messages instead of syslog in most default configurations — but the underlying logic is the same.

logrotate: Why Log Files Don't Grow Forever

A busy server can write hundreds of megabytes of logs per day. If those files grew without limit, disk space would eventually run out. logrotate — a periodically run tool that automatically rotates log files — solves this.

cat /etc/logrotate.d/rsyslog

/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

What this configuration means:

• daily — the file is rotated once a day;
• rotate 7 — only the last 7 rotated copies are kept; anything older is deleted;
• compress — old copies are compressed as .gz, saving disk space;
• missingok — do not error out if the file does not exist;
• notifempty — do not rotate an empty file, avoiding a pointless empty archive;
• the command inside postrotate tells the writing service that the file is fresh again, typically by sending it a signal.

In practice this looks like: when syslog is rotated, it is renamed to syslog.1, the previous syslog.1 is compressed into syslog.2.gz, and so on, up to the rotate 7 limit.

ls /var/log/syslog*

/var/log/syslog
/var/log/syslog.1
/var/log/syslog.2.gz
/var/log/syslog.3.gz

logrotate itself normally runs automatically through a daily cron job or a systemd timer — you don't have to trigger it by hand, but understanding its configuration is valuable when diagnosing problems such as a full disk.

postrotate vs copytruncate: How the Writer Keeps Up

The postrotate block above exposes the one hard problem in rotation: after the file is renamed to .1, the service is still writing to the old, renamed file until it is told the file is fresh. The postrotate signal (a HUP, or rsyslog-rotate here) is what makes it reopen the original path. A service that cannot be signaled to reopen its log needs the alternative copytruncate directive instead:

copytruncate is the standard answer for an application that holds its log file open and offers no reopen signal — at the cost of that small race window. Knowing why it exists, and its data-loss caveat, is a frequent interview follow-up to "how does log rotation work."

A rotation rule can be tested without waiting for the daily timer — a debug dry-run shows what would happen and changes nothing, while a force run performs it immediately:

logrotate -d /etc/logrotate.d/rsyslog        # dry-run: prints decisions, writes nothing
sudo logrotate -f /etc/logrotate.d/rsyslog   # force a rotation right now

Which Source to Check First

This table is the map the next two articles fill in: System Logs and dmesg goes deeper into kernel-level logging, and Log Analysis covers turning collected logs into practical conclusions.

Common Mistakes

Checking Only One Log Source and Stopping There

journalctl surfaces service-level failures clearly, but if the root cause sits in the kernel — a network driver, for instance — the answer only exists in dmesg. Any non-trivial problem deserves at least two sources checked together: the journal and the kernel log.

Trying to Read auth.log or wtmp with cat/less

wtmp, btmp, and lastlog are binary, not text. They must be read through their dedicated tools (last, lastb, lastlog); reading them directly just prints garbage to the screen.

Assuming rm on a Log File Frees Disk Space Immediately

As shown above, deleting a file that a process still has open does not reclaim disk space until that process closes the file or restarts. The correct approach is logrotate, or a service reload.

Exercises

1. From the output of ls -lh /var/log/, pick five different files and write, in your own words, what each one is for.
2. Open three files inside /etc/logrotate.d/ and compare their rotate, daily/weekly, and compress settings.
3. Run last -n 10 and, if permitted, sudo lastb -n 5, and compare the results against auth.log or journalctl -u ssh.service.
4. Pick three scenarios from the "Which Source to Check First" table above and write the exact command you would run for each.

Verification criterion: for each of the five layers introduced in this article, you should be able to name one concrete symptom that would send you specifically to that layer and no other.

References

• man7.org: syslog(3): https://man7.org/linux/man-pages/man3/syslog.3.html
• man7.org: logrotate(8): https://man7.org/linux/man-pages/man8/logrotate.8.html
• Ubuntu Server Documentation: Logs: https://ubuntu.com/server/docs/logs
• freedesktop.org: systemd-journald.service(8): https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html