Lab: Rolling Out SSH Hardening Without Locking Yourself Out

SSH Hardening laid out the full checklist and the reasoning behind each item. This lab is the hands-on rehearsal of applying that checklist in the correct, safe order on a disposable machine — including deliberately making the one mistake that locks people out (disabling password authentication before confirming key-based login actually works), so that the failure and its recovery are both familiar before they ever happen on a server that matters.

Goal and End Result

By the end of this lab, a lab VM will accept only key-based SSH authentication, reject direct root login, restrict logins to one named user, and you will have personally experienced — and recovered from — a lockout caused by disabling a fallback too early.

Topology and Starting State

Two terminal sessions to the same lab VM are required throughout this entire lab: a primary session, where changes are made, and a fallback session, kept open and untouched, that provides a way back in if a change breaks SSH access. This mirrors SSH Hardening's own repeated warning, and this lab is designed specifically to make skipping it hurt.

Requirements

• a lab VM reachable over SSH, ideally one with console access independent of SSH (a hypervisor console, or physical access) as a true last-resort fallback;
• an SSH key pair already generated on your client machine (see SSH Key Authentication if you need to generate one).

Security and Snapshot Note

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.lab-backup

Step 1: Confirm Key-Based Login Works — Before Touching Anything

ssh-copy-id -i ~/.ssh/id_ed25519.pub <username>@<vm-ip>

Verify, from a fresh session (not the one you just used to copy the key):

ssh -i ~/.ssh/id_ed25519 <username>@<vm-ip> "echo key auth works"

key auth works

Do not proceed until this exact verification succeeds. Every subsequent step in this lab assumes it did.

Step 2: Restrict Login to a Named User

sudo sed -i '/^AllowUsers/d' /etc/ssh/sshd_config
echo "AllowUsers <username>" | sudo tee -a /etc/ssh/sshd_config
sudo sshd -t

sshd -t syntax-checks the configuration without reloading it — always run this before restarting sshd, since a syntax error caught before a restart is a five-second fix, and the same error caught after a restart with a dropped connection is a much harder problem to recover from.

sudo systemctl reload sshd   # 'ssh' on Debian/Ubuntu — see the note above

Verify from a new session:

ssh <username>@<vm-ip> "whoami"

Keep the fallback session (Step 0's requirement) open throughout — do not close it yet, even though this step succeeded.

Step 3: Disable Direct root Login

sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl reload sshd

Verify:

ssh root@<vm-ip>

root@<vm-ip>: Permission denied (publickey).

Step 4: The Deliberate Mistake — Disable Passwords Before Verifying the Fallback

This step exists specifically to create the failure, safely, in a controlled setting.

sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl reload sshd

Verify the intended behavior first:

ssh -o PubkeyAuthentication=no -o PasswordAuthentication=yes <username>@<vm-ip>

Permission denied (publickey).

This confirms password authentication is correctly disabled — exactly as intended. Now the lab introduces the actual mistake: imagine the key copied in Step 1 was for the wrong user, or copied to the wrong path, and was never actually verified — close your fallback session now, on purpose, to feel the consequence:

# In the fallback session:
exit

# From your client machine, attempt a fresh connection:
ssh -i ~/.ssh/some-other-key <username>@<vm-ip>

Permission denied (publickey).

With no password authentication available and the wrong key presented, this connection genuinely fails — and if this were the only session, the machine would now be locked out from SSH entirely, recoverable only through console access.

Practical Scenario: Recovering from the Lockout

Problem. SSH access is refused, PasswordAuthentication is disabled, and the correct key is not being offered.

Check — first, confirm you are actually locked out, not just using the wrong key:

ssh -v -i ~/.ssh/id_ed25519 <username>@<vm-ip>

The -v flag shows exactly which keys were offered and why each was rejected — often the fix is simply specifying the correct key file explicitly, not a real lockout at all.

Command — recover via console access (the true fallback this lab has been building toward):

# Via the hypervisor/provider console, not SSH:
sudo sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sudo systemctl reload sshd

Verify:

ssh <username>@<vm-ip>

Result. Password authentication is restored, restoring access; the correct key can now be diagnosed and re-copied with ssh-copy-id before disabling password authentication again — this time only after re-confirming Step 1's verification exactly.

Conclusion. The entire point of keeping a fallback session open (Step 0) is to avoid ever needing this recovery path for real — this scenario exists so the failure mode and its console-based fix are already familiar, rather than being learned for the first time during an actual, higher-stakes incident.

Troubleshooting Tips

• sshd -t reports a syntax error after an edit. Fix it before reloading — never restart or reload sshd with a configuration that failed its own syntax check.
• AllowUsers blocks a legitimate second account you forgot about. AllowUsers is a space-separated list on one line, not a directive that can be repeated safely — check for exactly one AllowUsers line listing every intended user, since a second, later line silently overrides rather than adds to the first.
• Console access itself requires a password you no longer remember, since key-based login was the only method used for weeks. This is a real, recurring operational trap — periodically verify console/out-of-band access still works, not only the SSH path used daily.

Rollback and Cleanup

sudo cp /etc/ssh/sshd_config.lab-backup /etc/ssh/sshd_config
sudo sshd -t && sudo systemctl reload sshd
sudo rm -f /etc/ssh/sshd_config.lab-backup

Final Assessment Criterion

This lab is complete when all of the following hold:

• key-based login works and was verified from a fresh session before any restrictive change was made;
• PermitRootLogin no and PasswordAuthentication no are both confirmed active with a failed-login test, not just by reading the config file;
• you have personally triggered and recovered from a lockout using console access, and can describe the exact recovery command used;
• the original configuration is restored via the backup copy at the end of the lab.

Exercises

1. Repeat Steps 1–3 on a fresh lab VM, but this time also change the SSH port following SSH Hardening's guidance, and update the firewall rule accordingly before reloading sshd. Verify connectivity on the new port before closing any existing session.
2. Explain, in your own words, why sshd -t catching a syntax error before a reload is meaningfully safer than discovering the same error after the reload has already dropped active connections.
3. Configure fail2ban for sshd following SSH Hardening's coverage, then deliberately fail a login four times from a test client and confirm the client's IP is banned.

References

• OpenSSH: sshd_config(5)
• OpenSSH: sshd(8)
• man7.org: ssh-copy-id(1)