The Troubleshooting Process
Every tool covered in this module — Linux Logs, dmesg, Log Analysis, Resource Troubleshooting — does not solve a problem by itself. Each is a way to ask a question; what actually solves a problem is applying those tools in the right order, following a definite methodology. The real difference between an experienced administrator and a beginner is often not knowledge of commands — it's following a consistent process instead of a series of random attempts. This article presents that process in a general form, applicable to any type of problem: a service, the network, storage, or an application.
Why Random Attempts Don't Work
The most natural, but least effective, reaction when a problem shows up is to immediately try changing something: "maybe restarting the service will help," "maybe rebooting the server will fix it." This approach has three serious flaws:
- the problem can disappear by accident (a reboot clears a temporary memory leak, for instance), while the real cause remains and resurfaces later;
- if several changes are made at once, it becomes impossible to know which one actually helped;
- once the problem is gone, nobody can explain why it happened in the first place — which does nothing to prevent the exact same problem from recurring.
A systematic process is discipline against random attempts — not just to find the "right" answer faster, but to prove why it's right.
The Five-Step Process
flowchart TD
A["1. Define the problem precisely"] --> B["2. Gather evidence"]
B --> C["3. Form a hypothesis"]
C --> D["4. Test with one change"]
D --> E{"Resolved?"}
E -->|No| C
E -->|Yes| F["5. Document"]
Step 1: Define the Problem in a Precise, Testable Form
"The server is slow" is not a definition — it's an impression. A proper definition must be measurable and reproducible:
| Poor definition | Good definition |
|---|---|
| "The site is slow" | "The /api/orders endpoint is responding in 1.5 seconds instead of an average of 200ms, since 09:00" |
| "The service isn't working" | "backend-demo.service shows failed in systemctl status; the last successful start was at 11:05" |
| "The disk is full" | "df -h / shows 98%, up from 72% two days ago" |
A precise definition answers two questions: when it started, and how it's measured. Both form the foundation for every step that follows.
Step 2: Gather Evidence (Before a Hypothesis)
The single most important rule of this step is to change nothing yet — only observe and record the current state. Every tool covered in this module exists to serve exactly this step:
journalctl -u backend-demo.service --since "1 hour ago" --no-pager
dmesg -T | tail -n 30
vmstat 2 5
df -h /
All of these commands are read-only — they change nothing, they only capture a snapshot of the current state. The grep/awk techniques from Log Analysis and vmstat/iostat from Resource Troubleshooting are exactly the tools of this evidence-gathering step.
Tip
While gathering evidence, check at least two or three independent sources — for example, a service log and resource state — even when the cause seems obvious right away. A conclusion built on a single source is often incomplete, since, as Linux Logs explained, different layers answer different questions.
Step 3: Form a Hypothesis
Based on the evidence gathered, form a specific, testable guess in the shape of "I think the cause is X." A good hypothesis takes this form: "If the cause is X, I should observe Y."
For example: "If the slowdown is caused by disk I/O saturation, then iostat -x should show that disk's %util consistently above 90%."
This shape matters because it makes the hypothesis falsifiable — if %util turns out to be low, the hypothesis is wrong and a new one needs to be built, without wasting further time chasing it.
Step 4: Test with One Change
Once a hypothesis is confirmed, a fix is applied — but only one change at a time. If three things change at once — editing a configuration, restarting a service, and rebooting the server — then even if the problem disappears, there is no way to know which change actually fixed it. That, in turn, makes it impossible to reproduce the fix the next time the same problem shows up on a different server.
After the change, the result is checked immediately against the same measurable criterion defined in Step 1. If the hypothesis turns out to be wrong, the process returns to Step 3 — that is a normal outcome, not a failure of the process.
Step 5: Document
Once the problem is resolved, this last step is the one most often skipped — yet it's exactly what saves time in the future. A short write-up should include:
- how the problem first appeared (the initial symptom);
- what the actual root cause was;
- how it was fixed;
- what can be done to prevent this same problem in the future — adding monitoring, expanding a resource, fixing a configuration.
This record is the foundation of what's called root cause analysis (RCA). The point of RCA isn't to find someone to blame — it's to make the system more resilient against this class of problem.
Interview angle
"How do you actually find a root cause?" is a common interview question, and the simplest concrete technique is the Five Whys: keep asking "why" until you pass the symptom and reach something you can permanently fix. For the OOM scenario below: the service keeps restarting → why? it was killed by SIGKILL → why? the OOM killer chose it → why? it grew to 398 MB → why? a request handler never frees a buffer. The fix belongs at that last "why" — a code leak — not at the first, since restarting the service just repeats the loop. You have reached a root cause when the next "why" leaves your system's control (a genuine third-party bug, a hardware fault); anything above that is still a symptom.
Practical Scenario: The Full Process, End to End
Step 1. Definition: "backend-demo.service has been restarting every 5 minutes since 12:00 (via Restart=on-failure), even though curl usually returns a successful response."
Step 2. Evidence:
Jul 24 12:04:12 systemd[1]: backend-demo.service: Main process exited, code=killed, status=9/KILL
Jul 24 12:09:15 systemd[1]: backend-demo.service: Main process exited, code=killed, status=9/KILL
Jul 24 12:04:11 kernel: Out of memory: Killed process 8891 (python3) total-vm:412000kB, anon-rss:398200kB
Step 3. Hypothesis: "If the OOM killer is the cause, memory pressure should have been high during this window."
free is low, si/so are active — the hypothesis is confirmed. This is exactly the OOM mechanism covered in depth in Process Memory, showing up in this specific case.
Step 4. Test: either add a memory limit to the service (the MemoryMax= directive) or fix the memory leak in the application itself — one is chosen, not both at once, and the result is observed separately.
Step 5. Documentation: "backend-demo.service was being killed by the OOM killer every few minutes due to a memory leak. Temporary mitigation: Restart=on-failure already provides automatic recovery. Permanent fix: patch the leak in the application code, scheduled for next week."
Common Mistakes
Forming a Hypothesis Without Gathering Evidence
Jumping straight to a fix based on "this is usually caused by X," without checking, often wastes time chasing the wrong lead. Always gather at least one piece of evidence — a log, a resource snapshot — first.
Changing Several Things at Once
Editing configuration, code, and a resource limit all at the same time, then declaring "it worked," leaves no way to determine which change was the actual fix.
Not Documenting After the Problem Disappears
A reboot or a restart can make a problem "go away," but the underlying cause remains. A problem resolved without documentation often reappears, in the same shape, weeks or months later.
Exercises
- Pick a technical problem you encountered in the past week (or invent one) and write it out in both the "poor definition" and "good definition" style shown in the table above.
- Retell the practical scenario above step by step in your own words — explain which tool was used at each step and why in that particular order.
- On a test VM, deliberately break a unit (for example, with an invalid
ExecStart=as shown in Building a Custom systemd Service), work through all five steps, and write a short document at the end. - Explain, using a concrete example, why violating the "one change, one test" principle by making two changes at once is risky.
Verification criterion: your Step 5 write-up from exercise 3 must state the root cause in a form specific enough that a different administrator, reading only that document, could reproduce your fix without repeating your investigation from scratch.
Common Ground With Domain-Specific Troubleshooting
The five-step process above is deliberately generic — it does not depend on whether the problem is a service, a resource, storage, or the network. The domain-specific techniques covered elsewhere in this course, such as Network Diagnostics and Disk Full Troubleshooting, all supply the evidence-gathering and hypothesis-testing detail for Steps 2 through 4; the overall structure — define, gather, hypothesize, test one change, document — stays the same regardless of which domain the problem lives in.
References
- Google SRE Book: Effective Troubleshooting: https://sre.google/sre-book/effective-troubleshooting/
- Red Hat: Root Cause Analysis Basics: https://www.redhat.com/sysadmin/root-cause-analysis
- man7.org:
journalctl(1): https://man7.org/linux/man-pages/man1/journalctl.1.html