Cluster DNS and network policy

Two questions remain from the Service article. How does a pod turn the name api into 10.96.184.22? And given that the pod network is flat — every pod able to reach every other pod — what stops a compromised front-end pod from connecting straight to the database?

Both answers are cluster-wide policy objects, and both fail in ways that produce no error message.

Names in the cluster

kubectl exec -it redis-0 -- cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Three lines, each with consequences.

10.96.0.10 is the ClusterIP of the DNS Service, normally backed by CoreDNS pods. It's a Service like any other, which means cluster DNS is subject to everything from the previous article — including having no endpoints if its pods aren't ready.

The search list is why short names work. A Service's real name is <service>.<namespace>.svc.cluster.local, and a pod in the default namespace that queries api gets default.svc.cluster.local appended, producing api.default.svc.cluster.local, which resolves. The same query from a pod in the payments namespace appends payments.svc.cluster.local instead — and gets NXDOMAIN, because there's no api Service in payments.

A short name resolves relative to the pod's own namespace. Crossing namespaces requires at least <service>.<namespace>.

That's the answer to "it worked in dev and broke in the new namespace," and it's a standard interview question.

ndots:5 is the one with a performance cost. Any name containing fewer than five dots is treated as partial and tried against the search list first. Resolving api.example.com — two dots — means four failed queries before the real one:

api.example.com.default.svc.cluster.local   → NXDOMAIN
api.example.com.svc.cluster.local           → NXDOMAIN
api.example.com.cluster.local               → NXDOMAIN
api.example.com                             → 203.0.113.44

Four wasted round trips to CoreDNS for every external lookup, from every pod. On a busy cluster this is a genuine load source, and when CoreDNS is under pressure it turns into intermittent resolution failures for external services while in-cluster names stay fine — a confusing signature until you know the cause.

The fix is a trailing dot in the hostname (api.example.com.), which marks it absolute and skips the search list entirely, or a per-pod dnsConfig lowering ndots.

Headless Services return pod addresses

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
    - port: 6379

clusterIP: None disables the virtual IP entirely. There's no DNAT rule, no load balancing — DNS returns the pod addresses directly:

kubectl exec -it api-7d4b9c8f5-2xk4p -- nslookup redis

Name:   redis.default.svc.cluster.local
Address: 10.244.1.9
Name:   redis.default.svc.cluster.local
Address: 10.244.2.11

This exists because per-connection random balancing is wrong for some workloads. A database client that needs to address a specific replica, a StatefulSet member that must be reachable at a stable name, or a gRPC client doing its own load balancing all need the individual addresses rather than one virtual IP that pins them to a single backend. StatefulSet pods additionally get per-pod names — redis-0.redis.default.svc.cluster.local — which is how clustered databases find their peers.

Debugging DNS from inside

Minimal images have no dig or nslookup, which makes DNS debugging in a container annoying at exactly the wrong moment. Run a throwaway pod that does:

kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/agnhost:2.39 --restart=Never -- sh

Then apply the ladder from the DNS failures article, adjusted for the cluster:

nslookup kubernetes.default          # does in-cluster resolution work at all?
nslookup api.default.svc.cluster.local   # fully qualified, skipping the search list
nslookup api.example.com.            # external, trailing dot

If the first fails, CoreDNS itself is the problem — check its pods and its Service's endpoints. If the first works and the second doesn't, the Service doesn't exist under that name. If in-cluster names work and external ones don't, CoreDNS's upstream forwarding is the place to look, not anything about your application.

Everything can reach everything, until you say otherwise

This is the default nobody expects on first contact: a Kubernetes cluster with no NetworkPolicy allows every pod to open a connection to every other pod, in every namespace, on every port. The front-end can reach the database. A pod in staging can reach one in production on the same cluster. Namespaces are an organisational boundary, not a network boundary.

A NetworkPolicy changes that, with semantics worth reading twice:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 5432

podSelector picks which pods the policy governs — here, the database. policyTypes: [Ingress] says this policy is about incoming connections. The ingress block lists what's allowed: TCP 5432 from pods labelled app: api.

The rule that makes this work is the one to memorise:

A pod is unrestricted until at least one policy selects it. From then on, everything not explicitly allowed in that direction is denied.

So this single policy simultaneously permits the API pods and denies everything else — no deny rule was written, and none exists in the API. And the direction matters independently: this policy names only Ingress, so the database's outbound connections remain completely unrestricted.

Three details that bite in practice:

Selectors are namespace-scoped by default. podSelector: {matchLabels: {app: api}} means API pods in this policy's namespace. Allowing traffic from another namespace requires namespaceSelector, and forgetting it produces a policy that silently blocks the traffic you meant to permit.

Blocking DNS breaks everything. An egress policy that doesn't allow UDP 53 to the DNS Service leaves pods unable to resolve any name — which surfaces as every dependency failing at once, with errors that look nothing like a network policy problem. Every egress policy needs a DNS allowance:

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Your CNI plugin has to enforce policy. NetworkPolicy is an API, not an implementation. Calico, Cilium, and Weave enforce it; some CNI plugins — including flannel on its own — accept the object and enforce nothing at all. The policy shows up in kubectl get networkpolicy, looks correct, and does nothing. There's no warning.

kubectl get networkpolicy -n production

NAME           POD-SELECTOR   AGE
db-allow-api   app=postgres   3m

That output is identical whether the policy is being enforced or completely ignored. The only trustworthy verification is to test it:

kubectl run -it --rm probe --image=alpine --restart=Never -n production -- \
  sh -c 'nc -zv -w 3 postgres 5432'

nc: postgres (10.244.2.14:5432): Operation timed out

A timeout from a pod that shouldn't have access means enforcement is working. A succeeded means it isn't, and you have a policy that exists only as documentation.

Write and test a deny before you trust an allow

The failure mode of network policy is silent permissiveness — a policy that reads correctly, applies cleanly, and blocks nothing. Nobody notices, because everything keeps working. That's exactly what a compromised pod counts on.

After applying any policy, verify it from both sides: from a pod that should be allowed (it must still work) and from a pod that shouldn't be (it must fail). A policy verified only from the allowed side has proven nothing.

A default-deny baseline

Once enforcement is confirmed, the strongest and simplest starting point is denying all ingress in a namespace and adding back what's needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

An empty podSelector selects every pod in the namespace, and an empty ingress list allows nothing. This is the default-deny principle from the security module, expressed as a Kubernetes object — and applying it to a running namespace will break things immediately, which is why it belongs in a test environment first, with per-service allow policies written before it lands in production.

Practice

From a pod in one namespace, resolve a Service in another using the short name, then <service>.<namespace>, then the fully qualified name. Record which succeed.
Capture DNS traffic in a cluster while a pod resolves an external hostname, and count the queries. Then repeat with a trailing dot and count again.
Create a headless Service alongside a normal one for the same pods, and compare what nslookup returns for each. Explain which you'd use for a database cluster and why.
Apply the db-allow-api policy, then verify it from an allowed pod and a disallowed one. If the disallowed pod still connects, find out which CNI plugin your cluster runs and whether it enforces policy.
Write an egress policy for a single pod that permits only DNS and HTTPS to the internet. Apply it, confirm the pod can still resolve names and fetch an HTTPS URL, and confirm it can no longer reach another pod in the cluster.

Exercise 5 is the whole module in one object: DNS, egress, port matching, and the flat pod network all interacting. If your policy works on the first try, you understood every piece of it — and if the pod loses the ability to resolve anything, you've just reproduced the most common network policy outage there is.