Skip to content

Lab: Diagnosing "The Service Isn't Reachable"

Network Diagnostics introduced ping, traceroute, ss, and tcpdump individually. This lab uses all four together, in the order an experienced administrator actually reaches for them, to diagnose the single most common on-call page in existence: "I can't reach the service." The exercise deliberately introduces three different causes behind the same symptom — a wrong listening address (Case 1), a blocked firewall port (Case 2), and a name that resolves to the wrong host (Case 3) — because from the outside, all three look identical, and correctly distinguishing them is the actual skill being tested. A fourth possible cause, the process not running at all, is ruled out first in every case by the ps/ss check at the top of the sequence — the cheapest thing to confirm, and the one it is most embarrassing to have skipped.

Goal and End Result

By the end of this lab you will have diagnosed three separate, deliberately introduced causes of "connection refused" or "connection timed out" for the same service, using a consistent, layered diagnostic sequence that starts at the process and works outward to the network — rather than guessing at the firewall first because it is the most dramatic-sounding suspect.

Topology and Starting State

A single lab VM is enough; a second machine (or a container on the same host, treated as a separate network endpoint) makes the "from the outside" tests more realistic but is not required.

Requirements

  • nc (netcat) or python3 -m http.server as a simple test service;
  • ss, ping, curl, and ufw (or iptables), all present by default or installable via apt.

Security and Snapshot Note

This lab opens and closes a firewall port on a lab machine. Confirm current firewall state before changing anything, and restore it exactly at the end — an open port left behind by mistake in a lab is a real, if small, security exposure if that VM is ever reused or exposed later.

sudo ufw status verbose

Step 0: Start a Simple Test Service

python3 -m http.server 8080 --bind 127.0.0.1 &

This deliberately binds to 127.0.0.1 only — the first failure this lab will diagnose.

Case 1: The Service Is Bound to the Wrong Address

Symptom, from another machine (or curl using the VM's real IP, not localhost):

curl -v http://<vm-ip>:8080

curl: (7) Failed to connect to <vm-ip> port 8080: Connection refused

Diagnose — Step 1: is the process even running?

ps aux | grep "http.server"

farruxbek   5210  ...  python3 -m http.server 8080 --bind 127.0.0.1

The process is running — this rules out "the service crashed" and narrows the hypothesis.

Diagnose — Step 2: what is it actually listening on?

ss -tulpn | grep 8080

tcp   LISTEN  0   5   127.0.0.1:8080   0.0.0.0:*   users:(("python3",pid=5210,fd=3))

127.0.0.1:8080 — the local address column — is the answer: this process only accepts connections originating from the machine itself, never from the network, regardless of firewall state. This is precisely the check Network Diagnostics flags as the most commonly skipped step — administrators frequently jump straight to firewall rules for a symptom that a bind address explains completely.

Fix and verify:

kill %1
python3 -m http.server 8080 --bind 0.0.0.0 &
ss -tulpn | grep 8080

tcp   LISTEN  0   5   0.0.0.0:8080   0.0.0.0:*   users:(("python3",pid=5344,fd=3))

curl -v http://<vm-ip>:8080

< HTTP/1.0 200 OK

Case 2: A Firewall Rule Blocks the Port

sudo ufw deny 8080/tcp

Symptom:

curl -v --max-time 5 http://<vm-ip>:8080

curl: (28) Connection timed out after 5001 milliseconds

Diagnose. Notice the error text itself is the first clue: "Connection refused" and "Connection timed out" are different failures with different causes, worth distinguishing immediately rather than treating both as "network is broken":

  • Connection refused — a machine actively responded that nothing is listening on that port (no process bound, or a firewall rule explicitly rejecting rather than dropping).
  • Connection timed out — no response arrived at all within the wait period, typically because a firewall silently dropped the packet, or a route genuinely does not exist.
ss -tulpn | grep 8080

tcp   LISTEN  0   5   0.0.0.0:8080   0.0.0.0:*   users:(("python3",pid=5344,fd=3))

The process is confirmed listening on the correct address — this rules out Case 1's cause and points the investigation toward the network path itself.

sudo ufw status verbose

Status: active
To                         Action      From
--                         ------      ----
8080/tcp                   DENY        Anywhere

Fix and verify:

sudo ufw delete deny 8080/tcp
sudo ufw allow 8080/tcp
curl -v --max-time 5 http://<vm-ip>:8080

< HTTP/1.0 200 OK

Case 3: DNS Resolves to the Wrong Address

echo "127.0.0.1 test-service.lab" | sudo tee -a /etc/hosts

Symptom:

curl -v --max-time 5 http://test-service.lab:8080

curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused

Diagnose. Since the earlier fixes were rolled back to a working state before this case, the failure here is purely name resolution — confirm with the same layered approach, checking resolution before assuming the service itself is at fault again:

getent hosts test-service.lab

127.0.0.1       test-service.lab

The hostname resolves to 127.0.0.1, not the VM's real interface address — DNS Tools covers why /etc/hosts entries take priority over DNS by default and are a frequent, easy-to-forget source of exactly this class of confusing, machine-specific failure that does not reproduce on any other host.

Fix and verify:

sudo sed -i '/test-service.lab/d' /etc/hosts
getent hosts test-service.lab

Common Mistakes

  • Checking the firewall before checking whether the process is even running. As this lab's three cases show, the firewall is only one of at least three plausible causes for the same symptom — checking it first, out of order, wastes time on the two-thirds of cases where it is not the actual cause.
  • Treating "connection refused" and "connection timed out" as the same problem. As explained above, they point toward different layers of the stack and different fixes entirely.
  • Testing with curl localhost instead of the machine's real address. A service bound to 127.0.0.1 will falsely appear to work when tested this way, hiding exactly the Case 1 bug from being caught during initial testing.

Rollback and Cleanup

kill %1 2>/dev/null
sudo ufw delete allow 8080/tcp
sudo sed -i '/test-service.lab/d' /etc/hosts
sudo ufw status verbose

Confirm the final ufw status verbose output matches what you recorded before Step 0.

Final Assessment Criterion

This lab is complete when, for each of the three cases, you can state — without re-reading this lab — the specific command that identified the root cause and explain why the symptom looked identical across all three from a client's point of view.

Exercises

  1. Reproduce Case 2, but this time use ufw reject instead of ufw deny, and compare the client-side error message and response time against the deny case. Explain the difference between deny (silently drop) and reject (actively refuse) in terms of what a client observes.
  2. Using tcpdump, capture traffic on port 8080 while reproducing Case 2, and identify in the capture whether the SYN packet was ever acknowledged, dropped silently, or answered with a reset.
  3. Design and diagnose a fourth cause behind the same "can't reach the service" symptom that isn't covered above (a wrong port number in the client's request, or a route missing entirely, for example), applying the same layered diagnostic order: process, listening address, firewall, then name resolution.

References