Skip to content

Running certificates in production

Certificates expire. That sentence is the whole reason this article exists, because a TLS certificate is the only part of a running system with a hard deadline built into it, and the failure is total — not degraded performance, not a slow page, but every client refusing to connect at a timestamp you could have written down months earlier.

The PKI article explained what a certificate proves. This one is about the operational half: getting one, renewing it without anyone noticing, and finding out about a problem before your users do.

What's actually in the directory

A certificate is not one file. With Let's Encrypt and certbot, you get four, and choosing the wrong one is the most common misconfiguration there is:

sudo ls -l /etc/letsencrypt/live/example.com/

total 4
lrwxrwxrwx 1 root root  36 Apr 29 09:14 cert.pem -> ../../archive/example.com/cert1.pem
lrwxrwxrwx 1 root root  37 Apr 29 09:14 chain.pem -> ../../archive/example.com/chain1.pem
lrwxrwxrwx 1 root root  41 Apr 29 09:14 fullchain.pem -> ../../archive/example.com/fullchain1.pem
lrwxrwxrwx 1 root root  39 Apr 29 09:14 privkey.pem -> ../../archive/example.com/privkey1.pem
File Contents Use it for
cert.pem The leaf certificate alone Almost nothing — this is the trap
chain.pem The intermediate certificate(s) only Servers that want the chain as a separate directive
fullchain.pem Leaf + intermediates, in order This is what nginx and most servers want
privkey.pem The private key The key directive; never leaves the server

Point a server at cert.pem and it starts fine, serves traffic fine, and works in your browser — while failing for curl, for every language HTTP client, and for every other service that calls it. That's the missing-intermediate failure, and its cause is almost always this one-word choice.

Note that these are symlinks into archive/, and they get repointed on renewal. Copying fullchain.pem somewhere else "to keep things tidy" breaks renewal silently: the copy never updates, and the server keeps serving a certificate that expired weeks ago.

Getting a certificate

ACME is the protocol behind Let's Encrypt, and its job is to prove you control a domain before issuing anything for it. Two ways to prove it, and the choice has real consequences:

HTTP-01 — the CA fetches a token from http://<domain>/.well-known/acme-challenge/<token>. Simple, needs no credentials, and requires that port 80 is open to the internet and that the domain already resolves to this server. It cannot issue wildcards.

DNS-01 — you publish a TXT record the CA then queries. Works for wildcards, works for servers with no public HTTP at all, and requires an API credential for your DNS provider — which is a secret with a lot of power, so scope it to the one zone if the provider supports that.

sudo certbot --nginx -d example.com -d www.example.com

--nginx does three things: obtains the certificate over HTTP-01, edits the nginx configuration to use it, and installs a renewal hook that reloads nginx. Every name that clients will use has to be listed with its own -d — a certificate for example.com alone does not cover www.example.com, because validation checks the SAN list exactly.

Verify what you got, from outside, before believing it:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -ext subjectAltName

notBefore=Apr 29 09:14:32 2026 GMT
notAfter=Jul 28 09:14:31 2026 GMT
subject=CN = example.com
X509v3 Subject Alternative Name:
    DNS:example.com, DNS:www.example.com

Test against the staging environment first

Let's Encrypt enforces rate limits per registered domain, and a misconfigured loop can exhaust your week's issuance in minutes — leaving you unable to get a real certificate until the window resets. Add --dry-run when testing renewal, and --test-cert when testing first issuance; the latter uses the staging CA, whose certificates are untrusted by design (browsers will warn, which is correct and expected).

The private key is the whole security model

sudo ls -l /etc/letsencrypt/archive/example.com/privkey1.pem

-rw------- 1 root root 1704 Apr 29 09:14 privkey1.pem

0600, owned by root. Anyone who can read that file can impersonate your site to anyone who trusts the certificate — the encryption is still perfect, and it now protects an attacker's connection instead of yours.

Three habits follow from that. Never commit a key to version control, not even a private repository, and not even briefly — treat any key that has touched a repository as compromised and reissue. Never copy keys between environments; generate a separate one per host. And when a key is exposed, revoke the certificate rather than only replacing it, or the old one stays valid until its natural expiry:

sudo certbot revoke --cert-path /etc/letsencrypt/live/example.com/cert.pem --reason keyCompromise

Renewal, and the two ways it silently fails

Let's Encrypt certificates are valid for 90 days, and certbot renews when 30 days remain. Automation is installed for you:

systemctl list-timers 'certbot*'

NEXT                        LEFT      LAST                        PASSED     UNIT           ACTIVATES
Mon 2026-08-03 21:47:12 UTC 11h left  Mon 2026-08-03 03:12:44 UTC 6h ago     certbot.timer  certbot.service

Two things go wrong, and neither of them raises an error at the time:

The renewal itself starts failing. The DNS record moved, port 80 got firewalled, the plugin's credentials expired. certbot logs it and exits; nothing pages anyone. Catch it by actually testing the renewal path rather than assuming the timer means it works:

sudo certbot renew --dry-run

Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/example.com/fullchain.pem (success)

The renewal succeeds and the server keeps the old certificate. nginx reads certificates at startup and holds them in memory. A new fullchain.pem on disk changes nothing until a reload, so the server serves an expired certificate from a directory containing a perfectly valid one — a genuinely maddening thing to debug at 2 a.m. if you're checking the file instead of the connection.

The fix is a deploy hook, which runs only when a certificate was actually renewed:

sudo certbot renew --deploy-hook "systemctl reload nginx"

Stored per-certificate in /etc/letsencrypt/renewal/example.com.conf, so it applies on every future run. Use reload, not restart — a reload re-reads configuration and certificates while keeping existing connections alive; a restart drops them.

Check the certificate the server is serving, not the file it has on disk. Those are two different things, and the gap between them is where certificate outages live.

Monitoring, so the deadline never arrives unannounced

Certificate expiry is the most predictable outage in infrastructure, which makes not monitoring it hard to defend. openssl x509 -checkend exits non-zero when a certificate expires within the given number of seconds, which is all a check script needs:

#!/bin/bash
# Exit 1 if the served certificate expires within 21 days (1814400 seconds).
host="$1"
if echo | openssl s_client -connect "${host}:443" -servername "$host" 2>/dev/null \
   | openssl x509 -noout -checkend 1814400 >/dev/null; then
    echo "OK: ${host} certificate valid for at least 21 more days"
else
    echo "CRITICAL: ${host} certificate expires within 21 days (or could not be read)"
    exit 1
fi

OK: example.com certificate valid for at least 21 more days

Point it at the public hostname, not at a file path, and run it from outside the server. That tests the whole thing at once: the file, the reload, the right certificate being served for the right SNI name, and the chain — the same things a real client checks. A file-based check passes happily while users see an error.

Twenty-one days is a deliberate choice with Let's Encrypt's 90-day certificates: renewal starts at 30 days remaining, so an alert at 21 means renewal has already failed at least once and there's still a comfortable week to fix it by hand.

Mutual TLS, when the client must prove itself too

Ordinary TLS authenticates one side. The server proves who it is; the client stays anonymous and proves itself later with a password or a token, at the application layer.

mTLS moves that proof into the handshake: the server also demands a certificate from the client and refuses the connection outright if it doesn't validate. An unauthorised caller never reaches your application code, never appears in your access log as a 401, and never gets a chance to try a credential — which is why service meshes and internal APIs increasingly default to it.

In nginx it's two directives:

server {
    listen 443 ssl;
    server_name internal-api.example.com;

    ssl_certificate     /etc/letsencrypt/live/internal-api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/internal-api.example.com/privkey.pem;

    ssl_client_certificate /etc/nginx/certs/internal-ca.crt;
    ssl_verify_client on;
}

ssl_client_certificate names the CA whose signatures the server will accept from clients; ssl_verify_client on makes presenting a valid one mandatory. That CA is normally your own internal one, not a public authority — you want to authenticate your services, and a certificate from a public CA proves only that someone controls a domain name.

Testing it needs both halves of the client identity:

curl -v --cert client.crt --key client.key https://internal-api.example.com/health

Without them, the connection fails during the handshake:

curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required

The operational cost is real, and it's worth naming before you adopt it: now every client has a certificate that expires, and a client certificate expiring breaks the caller rather than the callee — which makes it much easier to miss. Automate client renewal with the same seriousness as server renewal, or mTLS turns one deadline into fifty.

Practice

  1. Issue a staging certificate with --test-cert for a domain you control, then connect with openssl s_client and identify from the output alone that it came from the staging CA.
  2. Deliberately misconfigure a server to use cert.pem instead of fullchain.pem. Confirm it works in a browser and fails with curl, then explain the discrepancy in two sentences.
  3. Write the expiry-check script above into your own monitoring, and verify it alerts correctly by pointing it at one of the public test endpoints that serve a deliberately expired certificate.
  4. Renew a certificate manually with certbot renew --force-renewal without reloading nginx, and confirm using openssl s_client that the served certificate still has the old dates. Then reload and confirm it changes.
  5. Set up mTLS between two containers with a private CA, then try the request without a client certificate and record the exact error. Compare that error to what a 401 from the application would look like, and say which one an attacker learns more from.

Exercise 4 is the one that changes how you debug. Once you've watched a server serve an expired certificate that was replaced on disk twenty minutes earlier, "check the file" stops being an acceptable answer to "is the certificate current?"

Certificates decide whether a client can trust the server it reached. They say nothing about which clients should have been able to reach it at all — a question that predates TLS by decades and that this module turns to next.

Sources