Skip to content

When name resolution is the problem

Half the incidents that get filed as network problems are DNS problems, and they're filed wrong because the symptom looks identical: a service that can't reach another service. The difference is that in a DNS failure, no packet was ever sent to the destination — the application gave up before it had an address to send anything to.

The resolution process article covered how a name becomes an address in theory. This one is about the four ways that goes wrong on a real Linux host, and the specific commands that tell them apart.

dig and your application don't ask the same question

This trips up experienced people, so get it out of the way first.

dig +short api.internal

10.20.0.44

curl https://api.internal/health

curl: (6) Could not resolve host: api.internal

Both commands ran on the same host, seconds apart, and they disagree. Neither is broken.

dig is a DNS tool. It builds a DNS query and sends it straight to a nameserver over UDP port 53. That's all it does.

An application resolves names through the C library's Name Service Switch, which consults sources in the order listed in /etc/nsswitch.conf:

grep ^hosts /etc/nsswitch.conf

hosts:          files mdns4_minimal [NOTFOUND=return] dns

files means /etc/hosts. dns means an actual DNS query. The bracketed directive means that if mdns4_minimal returns "not found," resolution stops there and never reaches dns — which is one real way a name that resolves fine with dig fails for everything else on the box.

So use the tool that answers the question your application is asking:

getent hosts api.internal

10.20.0.44      api.internal

getent hosts goes through NSS exactly as an application does — /etc/hosts first, then DNS, honouring every switch rule along the way. When dig and getent disagree, believe getent, and then find out why they differ. The answer is almost always a stale entry in /etc/hosts that somebody added during a migration two years ago:

grep api.internal /etc/hosts

10.20.0.9   api.internal

Which resolver is even being used

On modern Ubuntu, /etc/resolv.conf rarely names a real DNS server:

cat /etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search internal.example.com

127.0.0.53 is systemd-resolved, a local stub listening on loopback. Your queries go there, and it forwards them to the upstream servers it learned from DHCP or netplan. So the server in /etc/resolv.conf is not the server actually answering, and pointing dig at 127.0.0.53 tells you what the stub has cached, not what the authoritative source says.

To see the real upstreams and the per-interface configuration:

resolvectl status

Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS
Current DNS Server: 10.20.0.2
       DNS Servers: 10.20.0.2 10.20.0.3
        DNS Domain: internal.example.com

Now you can bypass the cache and ask the upstream directly, which is how you tell a stale local cache from a genuinely wrong record:

dig @10.20.0.2 +short api.internal

If the upstream returns the new address and your application still gets the old one, the stub's cache is stale:

sudo resolvectl flush-caches

That command is safe and takes effect immediately. It is also the one people reach for far too early — flush after you've confirmed a mismatch, not as a first move, or you'll destroy the evidence that proves the cache was the problem.

Stale answers have a source, and it's the TTL

A record was changed an hour ago. Some clients see the new address, some see the old one. Nothing is broken — you're watching a TTL expire.

dig api.example.com

;; ANSWER SECTION:
api.example.com.    3542    IN  A   203.0.113.44

3542 is the seconds remaining before this cached answer expires. Ask again in ten seconds and it will read 3532. A resolver that cached the old value will keep serving it for exactly that long, no matter what the authoritative server now says, and there is no mechanism to reach into other people's caches and correct them.

Which gives you a rule with real operational teeth:

Lower the TTL before you plan to change a record, not after. Drop it to 60 seconds a day ahead of a migration, make the change, verify, then raise it back. Changing a record that's still published with a 24-hour TTL means up to a day of split traffic, and no amount of flushing fixes it for anyone but you.

To see what the authoritative server says, skipping every cache in the path, ask it directly:

dig +short NS example.com

ns1.example-dns.net.
ns2.example-dns.net.

dig @ns1.example-dns.net api.example.com

;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
api.example.com.    3600    IN  A   203.0.113.44

The aa flag in the flags line means authoritative answer — this server owns the zone and isn't repeating something it cached. That's the ground truth, and any disagreement between it and what your host sees is a caching problem somewhere in between.

NXDOMAIN is cached too, and that surprises people

Create a DNS record, and it doesn't work — for a while — even though the record clearly exists now. What happened is that something queried the name before it existed, got NXDOMAIN, and cached that negative answer.

dig api.example.com

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41022
;; AUTHORITY SECTION:
example.com.    900 IN  SOA ns1.example-dns.net. admin.example.com. 2026080301 7200 3600 1209600 900

The status: NXDOMAIN says the name doesn't exist. The lifetime of that negative answer is the last field of the SOA record — 900 here — not the TTL you set on the record you just created. That's defined behaviour, not a quirk: negative caching uses the zone's SOA minimum field.

So "I created the record and it still doesn't resolve" has a specific answer: wait out the SOA minimum, or flush the caches you control. And it has a specific prevention: don't point clients at a name before the record exists.

Search domains, and why one lookup becomes five

That search internal.example.com line in resolv.conf isn't cosmetic. A name with no trailing dot and fewer dots than the ndots setting gets the search domains appended and tried in turn before the name is tried on its own.

This is invisible when it works and very visible when it doesn't. Inside Kubernetes, the default ndots is 5, and a pod resolving api.example.com — three dots, under the threshold — will first try api.example.com.<namespace>.svc.cluster.local, then api.example.com.svc.cluster.local, then api.example.com.cluster.local, and only then the name as written. Four queries fail before the fifth succeeds.

Every one of those adds latency, and if the cluster's DNS is under load, every one of them is a chance to time out. The symptom is external lookups that are slow or intermittently fail while in-cluster lookups are fine.

The fix is a trailing dot, which marks the name as fully qualified and skips the search list entirely:

dig api.example.com.

In application configuration, the same trailing dot in a hostname (https://api.example.com./) or a pod-level dnsConfig with a lower ndots does the same job.

A worked failure

A batch job on worker-03 started failing overnight with Could not resolve host: db.internal. Nothing was deployed. Run the ladder:

getent hosts db.internal

(no output, exit code 2)

Resolution genuinely fails the way the application sees it. Next, is it DNS specifically, or the whole resolver path?

dig +short db.internal

(no output)

dig @10.20.0.2 +short db.internal

;; communications error to 10.20.0.2#53: timed out

The upstream nameserver isn't answering at all. That reframes the incident completely: this isn't a record problem, it's a reachability problem to the DNS server — and the tools for that are the ones from the previous article.

nc -zv -u -w 3 10.20.0.2 53

nc: connect to 10.20.0.2 port 53 (udp) failed: Connection timed out

ip route get 10.20.0.2

10.20.0.2 via 10.20.5.1 dev eth1 src 10.20.5.9 uid 0
    cache

Traffic to the DNS server leaves via eth1 with a source address in 10.20.5.0/24. On the working hosts it leaves via eth0 from 10.20.0.0/24. Somebody added a route overnight, and the DNS server's firewall only permits queries from the original subnet.

Not one of those five commands was about DNS records. The DNS error was real; the DNS configuration was fine.

UDP makes DNS reachability tests slightly awkward

nc -zv -u reports success for a UDP port whenever nothing comes back with an ICMP port-unreachable, which for a filtered port looks the same as an open one. Treat a UDP "succeeded" as weak evidence. A dig against that server, which expects a real answer, is the stronger test — and DNS servers also listen on TCP/53, so nc -zv 10.20.0.2 53 gives you a definite TCP answer for the same host.

Practice

  1. Add an entry to /etc/hosts for a name that already resolves publicly, pointing it somewhere else. Then compare dig +short, getent hosts, and curl -v against that name and explain why they differ.
  2. Query a record with dig twice, thirty seconds apart, and watch the TTL count down. Then query the authoritative server for the same name and explain why its TTL doesn't count down.
  3. Find the SOA minimum for a domain you control with dig +short SOA <domain> and state how long a mistaken NXDOMAIN would persist in caches.
  4. On a host using systemd-resolved, find the real upstream DNS servers without reading /etc/resolv.conf.
  5. Write out the exact sequence of names a Kubernetes pod in namespace payments would query for redis and for api.example.com, given ndots:5. Then explain which of the two benefits from a trailing dot.

If exercise 1 produced three different behaviours, you've internalised the thing this article exists to teach: "DNS" is not one system, and the tool you test with decides which part of it you're testing.

Sources