Linux Production Deployment Security — A Practical Primer

#The Mental Model: Minimize, Control, Observe, Update

Everything below is one of four ideas:

MINIMIZE  → run as little as possible (fewer packages, fewer ports, fewer privileges)
CONTROL   → tightly govern who gets in and what they can do (access + authz)
OBSERVE   → log and audit everything so you can detect and reconstruct
UPDATE    → keep it patched and reproducible, without snowflakes

Memory hook

"a secure server is a boring, small, watched, fresh one." Boring = does one job. Small = minimal attack surface. Watched = logged and audited. Fresh = patched and rebuilt from code, not hand-tweaked over years. If a production decision doesn't make the box more boring/small/watched/fresh, question it.

#1. Logging In — How Humans (and Machines) Access the Box

#SSH: the front door

SSH is how you reach a Linux server, and it's the most-attacked service on the internet (bots hammer port 22 constantly). The non-negotiables:

Memory hook

"named users in, root never, keys only, password never." Root login + password auth is the combination every credential-stuffing bot prays for. Logging in as a named user and escalating with sudo is what gives you the "who did what" trail — root login erases identity because everyone is "root."

#Better than static keys: short-lived access

The modern pattern for fleets is not copying SSH public keys to every server's authorized_keys (which sprawls and is hard to revoke). Instead:

Memory hook

the best SSH port is no SSH port. Every open SSH port is attack surface and a key to manage. Brokered access (SSM Session Manager) or short-lived certs beat static keys, which beat passwords. Aim to remove the standing door, not just lock it.

#Privilege: sudo, least privilege, no shared accounts

#2. What Should (and Shouldn't) Be Running

#Run the minimum

Every running service is attack surface. The discipline:

ss -tulpn          # what's listening, and which process owns it
systemctl list-units --type=service --state=running   # what's running and why

Memory hook

"if you can't name why it's running, turn it off." The single highest-leverage hardening question is "what is listening on the network, and does it need to be?" Most breaches start with a service that was running, exposed, and forgotten. Attack surface you removed can't be exploited.

#Run services with least privilege (systemd hardening)

Modern Linux runs services under systemd, which can sandbox them heavily — most teams don't use this and should:

# /etc/systemd/system/myapp.service — a hardened unit
[Service]
User=myapp                    # never root
NoNewPrivileges=true          # the service can't gain privileges (blocks setuid escalation)
ProtectSystem=strict          # filesystem is read-only except explicitly allowed paths
ProtectHome=true              # no access to /home
PrivateTmp=true               # isolated /tmp (can't tamper with others' temp files)
CapabilityBoundingSet=        # drop ALL Linux capabilities (add back only what's needed)
SystemCallFilter=@system-service   # seccomp: only allow normal syscalls
RestrictAddressFamilies=AF_INET AF_INET6   # no weird socket families

Memory hook

assume the service will be popped, then contain it. Running a service as a non-root user with NoNewPrivileges, a read-only filesystem, dropped capabilities, and a seccomp filter means that even a full RCE in the app lands the attacker in a tiny, powerless box. This is the same "container is a process, not a boundary — so sandbox the process" idea, applied with systemd. It turns "app compromised" into "app compromised, and they can't do anything with it."

#Containers in production

If you deploy containers, the security rules rhyme:

• Non-root inside the container (USER directive), read-only root filesystem, drop capabilities, no --privileged, no docker socket mounted.
• Minimal base images (distroless / Alpine) — fewer packages = fewer CVEs = smaller attack surface.
• Scan images for CVEs (Trivy/Grype) in CI and at registry.
• Remember: a container is not a security boundary — it shares the host kernel. For hostile multi-tenant workloads, use stronger isolation (gVisor, Firecracker/microVMs).

#3. Auditing & Logging — So You Can See What Happened

You cannot respond to what you can't see. Production logging has three jobs: operational (is it healthy?), security (was it attacked?), and forensic (what exactly happened?).

#What to log

#auditd — the forensic backbone

auditd watches the kernel and records security-relevant events with the responsible user and timestamp. A few high-value rules:

# Watch sensitive files for any change (who modified /etc/passwd?)
-w /etc/passwd -p wa -k identity     # -p wa = alert on Write + Attribute-change; -k = searchable tag
-w /etc/sudoers -p wa -k privilege   # someone editing sudoers = granting themselves root → high signal
-w /etc/ssh/sshd_config -p wa -k sshd

# Record every command execution (execve) — powerful for IR
-a always,exit -F arch=b64 -S execve -k exec

# Watch for loading kernel modules (rootkit indicator)
-w /sbin/insmod -p x -k modules

Memory hook

auth.log tells you who got in; auditd tells you what they did. SSH/sudo logs answer "who logged in and escalated"; auditd answers "what files did they touch, what did they execute, what did they change." Together they reconstruct the timeline. If you run one extra thing on a production box for security, it's auditd with rules on identity files and execve.

#Ship logs OFF the box (the rule attackers hate)

Local logs are worthless if the attacker can edit them. The first thing a competent intruder does is clear /var/log. So:

• Forward logs in real time to a central, append-only store (SIEM, a logging account, an immutable bucket) the production box can write to but not modify or delete.
• This preserves evidence even if the host is wiped, and enables cross-host correlation and detection.

Memory hook

"logs that live only on the box die with the box." Centralized, immutable logging is what separates "we got owned and have no idea what happened" from "we have the full timeline." An attacker can rm -rf /var/log locally, but they can't reach back into your logging account. Ship logs off-host, make them append-only. (Ties to order of volatility and analyst OPSEC.)

#File integrity monitoring

Tools like AIDE take a cryptographic baseline of important files and alert on changes — catching a backdoored binary, a modified sshd, or a new SUID file. Tripwire/AIDE answers "did anything important change that shouldn't have?"

#4. Updating & Patching — Staying Fresh Without Snowflakes

#Patch, and patch fast

Most breaches exploit known vulnerabilities that had a patch available. So:

#Immutable infrastructure — the modern way to update

The old way: long-lived servers, patched in place for years, each one a unique "snowflake" nobody fully understands. The modern way is immutable infrastructure:

DON'T patch the running server.
DO  rebuild a fresh, patched image (AMI/container) from code,
    deploy it, and DESTROY the old one.

Memory hook

"cattle, not pets." Pets are named, hand-fed, nursed back to health when sick (patched in place forever). Cattle are numbered and replaced when they break (rebuild from image, destroy the old). Treating servers as cattle makes them reproducible, patchable, and — as a side effect — resistant to persistence, because an attacker's foothold evaporates at the next deploy. "Don't fix the server, replace it."

#Configuration as code

How the box is configured — packages, users, services, firewall — should live in code (Ansible, or baked into the image via Packer), not in someone's memory or ad-hoc commands. Benefits: reviewable, auditable, reproducible, and you can prove what should be there (so drift = suspicious).

#The Production Hardening Checklist

ACCESS
[ ] SSH: key-only, no root login, named users + sudo, AllowUsers/Groups
[ ] Prefer brokered access (SSM Session Manager) or SSH certs over static keys
[ ] No shared accounts; service accounts are non-login (nologin shell)
[ ] MFA on human access where feasible

MINIMIZE
[ ] One role per box/container; remove unused packages, tools, interpreters
[ ] Audit listening ports (ss -tulpn); bind to localhost unless must be public
[ ] Host firewall default-deny inbound (nftables/ufw); only required ports open
[ ] Services run non-root with systemd hardening (NoNewPrivileges, ProtectSystem, seccomp)
[ ] Containers: non-root, read-only FS, dropped caps, minimal base, scanned

OBSERVE
[ ] auditd with rules on identity files + execve
[ ] Logs shipped OFF-box in real time to append-only/central store
[ ] File integrity monitoring (AIDE)
[ ] Alerts on: root/sudo use, new SUID, new listening ports, config changes

UPDATE
[ ] Automated security patching (unattended-upgrades / dnf-automatic)
[ ] Fast lane for critical CVEs; know where vulnerable components run
[ ] Immutable infra: rebuild+replace, don't patch in place ("cattle not pets")
[ ] Config as code (Ansible/Packer/IaC), reviewed and version-controlled