System Logs and dmesg

Two of the four layers covered in Linux Logs — the kernel and syslog — are the oldest and lowest-level ones. Both predate systemd by decades, and both still run alongside the journal today. This article opens up those two layers in detail: the kernel's ring buffer and the rsyslog service — how each works, where it stores data, and how it is configured on a production server.

The Kernel Ring Buffer and `dmesg`

The kernel writes messages about itself — hardware detection, driver loading, memory-related warnings, and more — into a fixed-size ring buffer. It is called a "ring" because once the buffer fills up, the oldest messages are automatically overwritten by new ones; the buffer's size never changes.

dmesg | tail -n 5

[    4.812301] nvme0n1: p1 p2
[    5.023117] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode
[   12.441209] systemd[1]: Started Network Manager.
[  601.220304] usb 1-1: new high-speed USB device number 3 using xhci_hcd
[  601.331882] vfat: Unknown parameter 'utf8'

The number in brackets is the number of seconds elapsed since boot — not particularly convenient for a human trying to place an event in real time. To convert it to a readable date and time:

dmesg -T | tail -n 5

[Fri Jul 24 08:12:05 2026] nvme0n1: p1 p2
[Fri Jul 24 08:12:06 2026] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode
[Fri Jul 24 08:12:13 2026] systemd[1]: Started Network Manager.
[Fri Jul 24 08:22:01 2026] usb 1-1: new high-speed USB device number 3 using xhci_hcd
[Fri Jul 24 08:22:01 2026] vfat: Unknown parameter 'utf8'

Kernel Modules already used dmesg in its simplest form to check for module-loading errors. Here we cover the tool's full range of options.

Filtering by Severity

Kernel messages, like syslog messages, carry a severity level from 0 to 7 (see the table in the next section). To see only errors and anything more severe:

dmesg --level=err,crit,alert,emerg

Or, more briefly:

dmesg -l err

Watching Live

dmesg -w

-w (--follow) prints new kernel messages to the screen in real time, the same way journalctl -f does — useful for seeing exactly what happens the moment a USB device is plugged in or a disk error occurs. Stop it with Ctrl+C.

Info

The kernel ring buffer lives in memory and has a fixed capacity — if enough new messages arrive, the oldest ones disappear even without a reboot. For a durable, long-term history, use journalctl -k (kernel messages only, sourced from the journal) or /var/log/kern.log — unlike the ring buffer, both are written to disk.

journalctl -k -b

-k (--dmesg) restricts journalctl to kernel messages only, and -b restricts it to the current boot — in effect, this is dmesg's disk-backed version, delivered through the journal.

Syslog: Facility and Priority

rsyslog, and the classic syslog protocol it descends from, classifies every message along two dimensions:

facility — which part of the system the message originated from (kern, auth, daemon, mail, cron, local0–local7, and others);
priority (severity) — how serious the message is.

Level	Keyword	Meaning
0	emerg	System is unusable
1	alert	Action must be taken immediately
2	crit	Critical error
3	err	Ordinary error
4	warning	Warning condition
5	notice	Significant, but not an error
6	info	General informational message
7	debug	Detailed, developer-oriented information

This is the same table used by journalctl -p in journalctl — not a coincidence, since both are built on the same syslog standard (RFC 5424).

Interview angle

The local0 through local7 facilities exist specifically for custom applications that don't fit any predefined category — a common interview question is how you would route a homegrown script's output into the standard logging pipeline instead of writing your own log file handling. The answer is the logger command: logger -p local0.info -t myapp "message text" tags a message with a facility, a priority, and an identifying tag, then hands it to the syslog socket exactly as any other program would. From there it is subject to the same rsyslog selector rules as kernel or auth messages — no custom file-writing code required in the application itself.

`rsyslog` Configuration

The main configuration lives in /etc/rsyslog.conf, with additional configuration files under /etc/rsyslog.d/:

cat /etc/rsyslog.d/50-default.conf

auth,authpriv.*                /var/log/auth.log
*.*;auth,authpriv.none         -/var/log/syslog
kern.*                         -/var/log/kern.log
cron.*                         /var/log/cron.log

Each line follows the structure: facility.priority destination.

auth,authpriv.* — all severities from the auth or authpriv facility go to /var/log/auth.log;
*.*;auth,authpriv.none — every facility, every severity, except (none) auth/authpriv, goes to /var/log/syslog;
kern.* — all kernel messages go to /var/log/kern.log.

The - prefix before a destination path enables buffering instead of syncing to disk after every single write; this improves throughput, but means the last few entries can be lost in an unexpected power loss.

Adding a New Rule

For example, routing every message logged under the local0 facility to its own file:

sudo nano /etc/rsyslog.d/60-myapp.conf

local0.*    /var/log/myapp.log

sudo systemctl restart rsyslog

The configuration change only takes effect after rsyslog is restarted — the same pattern seen in Service Management.

Forwarding to a Central Log Server

rsyslog's original reason for existing on a fleet is that the same selector syntax can point at a remote host instead of a file — the basis of centralized logging, where every server ships its logs to one collector:

*.*    @central-logserver:514      # single @  = UDP  (light, may drop under load)
*.*    @@central-logserver:514     # double @@ = TCP  (reliable, ordered)

The single-@ (UDP) form is the classic interview trap: it is lighter but silently drops messages when the network or collector is saturated — exactly when logs matter most. Production central logging almost always uses @@ (TCP), often with RELP or TLS on top, precisely so that diagnostic evidence is not the first thing lost during an incident.

The Relationship Between `rsyslog` and `journald`

On Ubuntu Server, these two systems don't replace each other — they work together:

flowchart LR
    A["Kernel"] --> B["systemd-journald"]
    C["Applications / services"] --> B
    B --> D["Journal (binary format)"]
    B --> E["rsyslog (via the imjournal module)"]
    E --> F["/var/log/syslog, kern.log, auth.log ..."]

systemd-journald is the first to receive messages from every source — kernel, services, applications — and writes them into its own binary-format database, the mechanism covered in journalctl. rsyslog then reads from that same journal through its imjournal module and writes the messages back out into traditional, plain-text /var/log/ files.

Tip

In practice this means a single error message can be found both with journalctl -u nginx.service and with grep nginx /var/log/syslog — both are correct, they just differ in format and the tool used to search. journalctl is more convenient for fast, unit-scoped queries; plain-text syslog files are more convenient when working with generic text-processing tools (grep, awk, or forwarding to a central log server).

Practical Scenario: Diagnosing a Newly Connected USB Drive

Suppose an external drive is plugged into a server but does not show up as expected. Investigate using read-only commands only:

dmesg -T -w

Plug in the drive and watch for new lines — typically something resembling this:

[Fri Jul 24 12:40:11 2026] usb 1-1: new high-speed USB device number 5 using xhci_hcd
[Fri Jul 24 12:40:11 2026] usb-storage 1-1:1.0: USB Mass Storage device detected
[Fri Jul 24 12:40:12 2026] sd 4:0:0:0: [sdb] Attached SCSI disk

If the drive is not detected at all, no message appears — pointing to a hardware or cable problem. If [sdb] Attached SCSI disk does appear but the drive is still not mounted, the problem is no longer at the kernel level; it has moved into mount configuration, a separate topic covered elsewhere.

Common Mistakes

Reading `dmesg` Output Without Timestamps

Plain dmesg only shows seconds elapsed since boot, which makes it hard to immediately place a message on a calendar timeline. dmesg -T is almost always the better default.

Assuming the Ring Buffer Is Written to Disk

The ring buffer is memory-only — if enough new messages arrive, old data can vanish even without a reboot. For durable analysis, journalctl -k or /var/log/kern.log is more reliable.

Editing `rsyslog.conf` and Forgetting to Restart

Even a correctly written configuration file has no effect until the rsyslog service is reloaded; until then, it keeps operating on the old rules.

Expecting `dmesg` to Work Without `sudo`

On Ubuntu and most hardened kernels, kernel.dmesg_restrict is set to 1 by default, so a non-root user gets dmesg: read kernel buffer failed: Operation not permitted rather than output. Run dmesg (and dmesg -T/-w) with sudo, or read the same data from the disk-backed journalctl -k, whose access is governed by group membership (adm/systemd-journal) instead. The restriction exists because raw kernel-buffer contents can leak memory addresses and hardware details useful to an attacker.

Exercises

Run dmesg -T | tail -n 20, pick at least three messages, and explain what each one is about.
Open /etc/rsyslog.d/50-default.conf (or the equivalent file on your system) and analyze at least three rules in the form facility.priority → destination.
Using the mermaid diagram above, describe in your own words the path a single error message takes from the kernel to journalctl and to /var/log/kern.log.
On a test VM, add a new rule for the local0 facility, send a test message with logger -p local0.info "test message", and find it in the file you created.

Verification criterion: your rule from exercise 4 should route the test message only into the new file, not also into /var/log/syslog — if it appears in both, review the .none exclusion pattern used in the default configuration.

References

man7.org: dmesg(1): https://man7.org/linux/man-pages/man1/dmesg.1.html
man7.org: syslog(3): https://man7.org/linux/man-pages/man3/syslog.3.html
Rsyslog Documentation: https://www.rsyslog.com/doc/
RFC 5424: The Syslog Protocol: https://www.rfc-editor.org/rfc/rfc5424
man7.org: journalctl(1) — the -k flag: https://man7.org/linux/man-pages/man1/journalctl.1.html