Skip to content

Log Analysis

Linux Logs and System Logs and dmesg covered where logs live and how they get collected. The next step is turning that raw data into a practical conclusion. Skimming a single log file by eye is easy; a production server writing several million lines a day is not something anyone reads line by line. This article applies the tools already covered in Searching Files and Text and Rewriting Text: sed and awk specifically to log analysis, and adds journalctl's structured output formats to the toolkit.

Lab Data

The following commands build a test file that mimics a real production log, but stays fully under your control:

mkdir -p ~/lab/logs && cd ~/lab/logs
cat > app.log <<'EOF'
2026-07-24 09:12:01 INFO  ip=10.0.0.5 path=/api/login status=200 duration=42ms
2026-07-24 09:12:03 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1381ms
2026-07-24 09:12:05 INFO  ip=10.0.0.5 path=/api/orders status=200 duration=88ms
2026-07-24 09:12:07 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1290ms
2026-07-24 09:12:10 WARN  ip=10.0.0.7 path=/api/login status=401 duration=15ms
2026-07-24 09:12:12 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1455ms
2026-07-24 09:12:20 INFO  ip=10.0.0.5 path=/api/profile status=200 duration=30ms
EOF

Every command below runs against this ~/lab/logs/app.log file and is read-only — it has no effect on any real system.

Extracting the Right Events with grep

grep ERROR app.log

2026-07-24 09:12:03 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1381ms
2026-07-24 09:12:07 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1290ms
2026-07-24 09:12:12 ERROR ip=10.0.0.9 path=/api/orders status=500 duration=1455ms

To search for several conditions at once, use extended regular expressions (-E):

grep -E 'ERROR|WARN' app.log

Counting matches:

grep -c ERROR app.log

3

Viewing surrounding context — what happened just before and after an error line:

grep -B1 -A1 ERROR app.log

-B1 shows one line before the match, -A1 one line after — useful for understanding the event's surrounding context.

Searching Rotated and Compressed Logs

Real incidents rarely stay inside the current file. As Linux Logs showed, logrotate renames older logs to syslog.1 and compresses everything beyond that to syslog.2.gz, syslog.3.gz, and so on — and plain grep cannot read a .gz file. The z-prefixed variants decompress on the fly:

zgrep ERROR /var/log/syslog.2.gz
zcat /var/log/syslog.3.gz | awk '/ERROR/ { print }'

To sweep the entire rotated set — current file plus every archive — in one pass, zgrep accepts the glob directly and reads plain and compressed files together:

zgrep -h 'status=500' /var/log/syslog*

Forgetting this is a classic mistake: grep ERROR /var/log/syslog finds nothing from before the last rotation and quietly gives an incomplete picture, when the evidence you need is sitting one .gz file over.

Extracting Columnar Data with awk

Pulling only the ip= value out of every ERROR line:

grep ERROR app.log | awk -F'ip=' '{print $2}' | awk '{print $1}'

10.0.0.9
10.0.0.9
10.0.0.9

This two-stage approach is clear, but awk alone can do the same job in one command:

awk '/ERROR/ { for (i=1; i<=NF; i++) if ($i ~ /^ip=/) print $i }' app.log

Finding which IP address produced the most errors — one of the most common patterns in practice:

grep ERROR app.log | grep -oE 'ip=[0-9.]+' | sort | uniq -c | sort -rn

3 ip=10.0.0.9

The chain works as follows: grep -oE extracts only the matching fragment (ip=...), sort groups identical values together, uniq -c counts how many times each one repeats, and the final sort -rn orders the result from highest count to lowest — the most frequent value ends up first.

Analyzing response time (duration) — for example, finding the three slowest requests:

grep -oE 'duration=[0-9]+ms' app.log | grep -oE '[0-9]+' | sort -rn | head -n 3

1455
1381
1290

Tip

awk has no built-in sort, so numeric sorting is normally done by extracting the values with awk and finishing the job with sort -n — chaining the two tools together is a standard pattern, not a workaround.

Getting journalctl in Structured Form

A grep/awk chain over plain text requires knowing the log format in advance. journalctl entries are already structured, and can be pulled out in JSON:

journalctl -u ssh.service -n 3 --output=json-pretty

{
    "__CURSOR" : "s=8f2a...",
    "_SYSTEMD_UNIT" : "ssh.service",
    "PRIORITY" : "6",
    "MESSAGE" : "Accepted publickey for ali from 10.0.0.5 port 51422",
    "__REALTIME_TIMESTAMP" : "1753349532000000"
}

This format is convenient for scripted processing, since the fields (MESSAGE, PRIORITY, _SYSTEMD_UNIT) are known in advance and can be referenced without worrying about format changes. If jq is installed, for example:

journalctl -u ssh.service -n 50 --output=json | jq -r 'select(.PRIORITY|tonumber <= 3) | .MESSAGE'

This prints only the message text of entries at err severity or more severe (PRIORITY <= 3, matching the table in System Logs and dmesg).

Correlating Multiple Sources by Time

Diagnosing a real problem often takes more than one source. For instance, if there's an application error (ERROR in app.log), it helps to know what else was happening on the system at that exact moment:

journalctl --since "2026-07-24 09:12:00" --until "2026-07-24 09:12:15" --no-pager

This applies the --since/--until filter, covered in journalctl, to the exact window when an error occurred. If a systemd message about memory pressure or a service restart shows up in that window, it may well explain the real cause of the application error — resource-related situations like this are covered in more depth in Resource Troubleshooting.

Practical Scenario: Which Endpoint Is Causing the Most Trouble

awk '$3 ~ /ERROR|WARN/ { for (i=1;i<=NF;i++) if ($i ~ /^path=/) print $i }' app.log \
    | sort | uniq -c | sort -rn

3 path=/api/orders
1 path=/api/login

Interpretation: /api/orders is logged with three ERROR entries, /api/login with one WARN — so attention should go to /api/orders first. Rather than accepting this conclusion blindly, it is worth cross-checking the automatically computed count against the source file:

grep 'path=/api/orders' app.log

This verification step confirms the automated count against the raw log — a step that should never be skipped in professional analysis.

Interview angle

A common interview probe is: "how do you know your grep/awk chain isn't silently dropping or double-counting lines?" The honest answer is that any pipeline built from independent tools can behave unexpectedly on malformed input — a missing field, an unescaped character, an encoding issue. That is exactly why the verification step above (re-checking a computed count against grep on the raw source) is not optional polish; it is the difference between a number you can act on and a number that merely looks convincing.

Common Mistakes

Not Escaping Regex Special Characters

In grep '10.0.0.9', the dot (.) means "any character" in a regular expression, not a literal dot. To search for an exact IP address, use grep -F '10.0.0.9' (literal search) or grep '10\.0\.0\.9' (escaped) — otherwise a string like 10a0b0c9 could match when it shouldn't.

Not Verifying Computed Statistics Against the Source

A sort | uniq -c chain with correct syntax but the wrong column or filter can produce a wrong, yet confident-looking, number. Always cross-check an important conclusion against the raw log at least once before acting on it.

Treating journalctl's Plain-Text Format as Stable for Parsing

journalctl's default (text) output can shift slightly in formatting between versions. For anything a script depends on, explicit formats like --output=json or --output=cat are far more reliable than parsing plain text with awk/sed.

Exercises

  1. In the lab file above, use grep to extract all lines with status=500 and count them with -c.
  2. Using awk, extract the duration value from each line and compute the average (using a sum/count variable pattern in awk).
  3. Run journalctl -u ssh.service -n 20 --output=json-pretty and locate at least four fields (MESSAGE, PRIORITY, _PID, __REALTIME_TIMESTAMP).
  4. Invent your own scenario (for example, "which user produced the most 401 errors") and solve it against the lab file above using a grep/awk chain.

Verification criterion: for exercise 1, your computed count must match the number of lines you get from independently reading the raw file — if it doesn't, the filter condition is wrong, not the data.

References

  • man7.org: grep(1): https://man7.org/linux/man-pages/man1/grep.1.html
  • GNU Awk User's Guide: https://www.gnu.org/software/gawk/manual/gawk.html
  • man7.org: journalctl(1) — --output formats: https://man7.org/linux/man-pages/man1/journalctl.1.html
  • jq Manual: https://jqlang.org/manual/