Skip to content

tcpdump

Wireshark is the better tool for looking at a capture. It is not an option for taking one on a production server: there's no display, you're on SSH, and installing a GUI stack with hundreds of megabytes of dependencies onto a machine that's currently on fire is not a plan.

tcpdump is the capture tool that's already there. It's a few hundred kilobytes, it ships with nearly every Linux distribution, and it uses the same libpcap filter syntax and the same file format Wireshark reads — so the standard workflow is capture with tcpdump on the server, analyse with Wireshark on your laptop.

Why capturing needs root, and what that means

sudo tcpdump -i eth0 -nn -c 5 'tcp port 443'

Reading every frame that touches a NIC requires putting the interface into promiscuous mode and opening a raw packet socket. Ordinary users can't do either, which is why sudo is mandatory and why running tcpdump is a genuinely privileged act: a capture on a busy server can contain session cookies, API tokens, and any plaintext credential that crosses the wire.

A capture file is a credentials file until proven otherwise

Anything unencrypted in the capture is readable by anyone who later gets the .pcap. Write captures to a directory only root can read, delete them when the investigation ends, and never attach one to a ticket without checking what's in it. Filter narrowly — capture the one host and port you're investigating, not everything — both to protect data and to keep the file small enough to be useful.

On a busy interface, an unfiltered tcpdump can also fill a disk in minutes and cause a second outage on top of the one you're debugging. Always bound it with -c (a packet count) or -W/-C (rotating size-limited files).

The five flags that matter

Flag Effect Why you want it
-i eth0 Capture on this interface -i any captures on all of them, useful when you don't yet know which one traffic uses
-nn Don't resolve addresses or port numbers to names A single -n skips DNS lookups; the second n skips the ports-to-service-names translation. Without it, tcpdump issues its own DNS queries, which then show up in your capture
-c 20 Stop after 20 packets Bounded output on an interface doing thousands of packets a second
-w capture.pcap Write raw packets to a file instead of printing The file to open in Wireshark
-v, -vv More protocol detail per packet Adds TTL, IP ID, and header-length fields to each line

Note what -nn prevents. tcpdump resolving 142.250.185.78 means sending a DNS query, which generates packets, which your capture then records — a feedback loop that has confused people into thinking their server has a DNS problem when the capture tool created it.

Reading the output

sudo tcpdump -i eth0 -nn -c 5 'tcp port 443'

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:31:07.412885 IP 192.168.1.10.47122 > 142.250.185.78.443: Flags [S], seq 2847193021, win 64240, options [mss 1460,sackOK,TS val 3921847 ecr 0,nop,wscale 7], length 0
10:31:07.446231 IP 142.250.185.78.443 > 192.168.1.10.47122: Flags [S.], seq 1029384756, ack 2847193022, win 65535, options [mss 1412,sackOK,TS val 882913 ecr 3921847,nop,wscale 8], length 0
10:31:07.446301 IP 192.168.1.10.47122 > 142.250.185.78.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3921880 ecr 882913], length 0
10:31:07.447102 IP 192.168.1.10.47122 > 142.250.185.78.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3921881 ecr 882913], length 517
10:31:07.481003 IP 142.250.185.78.443 > 192.168.1.10.47122: Flags [.], ack 518, win 66, options [nop,nop,TS val 882948 ecr 3921881], length 0
5 packets captured
6 packets received by filter
0 packets dropped by kernel

Timestamps, sequence numbers, and addresses vary every run. The structure doesn't:

10:31:07.412885 — microsecond timestamp. The gaps between these are the measurement. Here the SYN goes out at .412885 and the SYN-ACK returns at .446231: a 33 ms round trip, which tells you the distance to the server before any application code has run.

192.168.1.10.47122 > 142.250.185.78.443 — source and destination, with the port appended after the final dot. Reading .47122 as part of the address is the classic beginner mistake; it's the ephemeral port the client picked.

Flags [S] — the TCP flag field, in tcpdump's compact notation, and the single most important thing to be able to read at a glance:

Notation Flags set Meaning
[S] SYN Opening a connection
[S.] SYN + ACK The server accepted it
[.] ACK only A bare acknowledgement
[P.] PSH + ACK Carrying application data
[F.] FIN + ACK Graceful close beginning
[R] or [R.] RST Connection refused or torn down abruptly

The . is always ACK. The first three lines above are the three-way handshake — [S], [S.], [.] — and if you can spot that pattern in a capture, you can answer "did the connection ever establish?" without reading anything else.

length 517 on the fourth packet is the first application data, and it's the TLS ClientHello. mss 1412 in the server's reply is worth noticing too: below the usual 1460, which means something in the path is encapsulating traffic.

The trailer matters as well. 0 packets dropped by kernel confirms the capture is complete. A non-zero number there means tcpdump couldn't keep up and your capture has holes — at which point conclusions drawn from "I don't see the packet" become unsafe.

Filters: capture less, understand more

The expression in quotes is a pcap filter, evaluated in the kernel before packets are copied to tcpdump. That's why filtering is cheap and why it's the first thing to reach for rather than piping through grep.

sudo tcpdump -i any -nn 'host 10.20.0.31 and port 5432'
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-rst != 0'
sudo tcpdump -i eth0 -nn 'icmp'
sudo tcpdump -i eth0 -nn 'net 192.168.1.0/24 and not port 22'

The not port 22 idiom in the last one is a habit worth forming immediately: without it, a capture taken over SSH records your own SSH session, which then generates more traffic, which gets captured. The output scrolls forever and none of it is what you're looking for.

The second filter — matching packets with the RST bit set — is the one that ends arguments. It answers "is something actively refusing our connections?" with a yes or a silence.

Investigation: the API call that fails only in production

Staging works. Production returns a connection error from the application, intermittently, maybe one request in twenty. The application logs say connection reset by peer and nothing else.

Start by asking who is sending the reset:

sudo tcpdump -i any -nn 'host 10.20.0.44 and tcp port 8443' -c 200 -w /root/reset-hunt.pcap

Then read it back — -r replays a saved file and takes the same filters:

sudo tcpdump -nn -r /root/reset-hunt.pcap 'tcp[tcpflags] & tcp-rst != 0'

reading from file /root/reset-hunt.pcap, link-type LINUX_SLL2 (Linux cooked v2)
10:47:22.104881 IP 10.20.0.44.8443 > 10.20.0.12.39114: Flags [R.], seq 1, ack 518, win 0, length 0
10:47:29.551204 IP 10.20.0.44.8443 > 10.20.0.12.39120: Flags [R.], seq 1, ack 518, win 0, length 0

Two things are now settled that were guesses before. The reset comes from the server, not from a middlebox on the way — and it arrives after ack 518, meaning the server accepted the connection, received 517 bytes of ClientHello, and then refused. A firewall dropping traffic wouldn't produce this; it would produce silence. A network problem wouldn't produce a well-formed RST from the destination address.

That narrows the investigation from "the network" to "the service on 10.20.0.44 is rejecting some TLS handshakes" — a different team, a different log file, and roughly a tenth of the search space. Which specific handshakes and why is a question for the TLS material, but the capture is what made it the right question.

Handing the file to Wireshark

sudo tcpdump -i eth0 -nn -c 2000 -w /root/incident.pcap 'host 10.20.0.44'
sudo chown $USER /root/incident.pcap

Then copy it down and open it locally:

scp <user>@<server>:/root/incident.pcap .

Wireshark's Follow TCP Stream and its protocol dissectors do things tcpdump's terminal output can't, and there's no reason to squint at hex on a server when the file is three megabytes. Capture where the traffic is; analyse where the screen is.

Practice

  1. Capture your own DNS traffic with sudo tcpdump -i any -nn port 53 -c 10 while running a dig in another terminal. Identify the query and the response, and measure the time between them from the timestamps.
  2. Capture a full connection to a website with -c 20, then find the three handshake packets by their flag notation alone, without reading the sequence numbers.
  3. Run a capture without -nn and then with it, both filtered to a single host. Explain what appears in the first capture that isn't real traffic.
  4. Deliberately connect to a closed port on your own machine while capturing, and identify the exact packet that carries the refusal. Then predict what the capture would look like if a firewall were dropping the packet instead — and verify it if you have a lab firewall to test against.

That last prediction is the bridge to everything ahead. A refused connection and a dropped connection look identical from an application's error message and completely different on the wire, and being able to tell them apart from a capture is what separates guessing at a network problem from diagnosing one.

Sources