Kubernetes Fundamentals

#Architecture Overview

┌──────────────── Control Plane ─────────────────┐
│  kube-apiserver  ←─── only entry point to state │
│  etcd            ←─── all cluster state stored  │
│  kube-scheduler  ←─── assigns pods to nodes     │
│  controller-mgr  ←─── reconciliation loops      │
│  cloud-controller←─── cloud-provider integration│
└─────────────────────────────────────────────────┘
            ↕  kubelet + kube-proxy
┌──────── Worker Node ────────────────────────────┐
│  kubelet          ←─── node agent, talks to API │
│  kube-proxy       ←─── iptables/ipvs rules      │
│  container runtime (containerd / CRI-O)         │
│  Pods                                           │
└─────────────────────────────────────────────────┘

etcd — distributed key-value store; contains all secrets, configs, and resource state. If compromised = full cluster compromise.

kube-apiserver — all kubectl commands hit the API server. Authentication → Authorization (RBAC) → Admission Control → etcd.

Memory hook

everything is "desired state in etcd, reconciled by controllers." The one idea that makes Kubernetes click: you don't run things, you declare what you want ("3 replicas of this image") and store that in etcd via the API server. Controllers are loops that constantly compare desired state (etcd) to actual state (the nodes) and nudge reality toward the declaration. Kill a pod and the Deployment controller notices the gap and makes a new one. So K8s is a declarative reconciliation engine, not an imperative runner — which is why GitOps works (git = desired state), why kubectl delete pod doesn't really "fix" anything (it gets recreated), and why the API server + etcd are the crown jewels.

#The Big Picture: How Kubernetes Is Actually Used End-to-End

Architecture diagrams hide the part people actually live in: the workflow from a developer's code to a running, monitored, secured workload. Here's the whole pipeline.

 ┌─ DEVELOPER ─┐   ┌──── CI (build & check) ────┐   ┌─ REGISTRY ─┐   ┌── GitOps ──┐   ┌──── CLUSTER ────┐
 │ writes code │   │ build image (Dockerfile)   │   │ store the  │   │ ArgoCD/Flux│   │ kube-apiserver  │
 │ + a Dockerfile│ │ scan image (Trivy/Grype)   │──▶│ signed     │   │ watches the│──▶│ admission checks│
 │ + k8s manifests│ │ run tests, SAST, IaC scan │   │ image      │   │ git repo & │   │ schedules pods  │
 │ → git push   │──▶│ sign image (cosign)        │   │ (ECR/GCR/  │   │ syncs to   │   │ runtime (Falco) │
 └──────────────┘   │ push manifest change to git│   │  GHCR)     │   │ the cluster│   │ + service mesh  │
        │           └────────────────────────────┘   └────────────┘   └────────────┘   └─────────────────┘
        │                                                                                       │
        └──────────────── observability: metrics (Prometheus), logs, traces, alerts ◀──────────┘

#The two deployment models: "push" vs "pull" (GitOps)

Memory hook

GitOps = "git is the desired state; an in-cluster robot makes reality match." Because K8s is already a reconciliation engine, GitOps just extends the loop out to git: you git push a manifest change, and ArgoCD/Flux (running inside the cluster) pulls it and applies it. Huge security wins: the cluster credentials never leave the cluster (CI can't be the breach path to prod), every deploy is a signed/reviewed git commit (audit + rollback = git revert), and config drift becomes detectable — if the live cluster differs from git, that's either an unauthorized change or an incident. This is how most modern shops deploy, and "do you use GitOps?" is a common interview question.

#What lives in the GitHub repo

A typical app repo (or a separate "infra"/"deploy" repo) contains:

my-service/
├── src/                      # application code
├── Dockerfile                # how to build the container image
├── .github/workflows/ci.yml  # build, scan, sign, push, bump manifest
└── deploy/
    ├── base/                 # Kubernetes manifests (Deployment, Service, etc.)
    │   ├── deployment.yaml
    │   └── service.yaml
    └── overlays/             # per-environment patches (dev/staging/prod)
        ├── staging/
        └── prod/             # ArgoCD watches this path

• Manifests are YAML describing the desired objects. Raw YAML doesn't scale across environments, so people template it:
  Helm

  package manager; templated charts with values per environment (values-prod.yaml). Think "npm for K8s."

  Kustomize

  patch/overlay model (no templating language); built into kubectl. A base/ plus environment overlays/.

#Core Workload Objects

#Networking

#Pod-to-Pod

• Every Pod gets a unique cluster IP
• All pods can reach all other pods by default (flat network)
• Network Policies restrict this

#Service Types

#DNS

• <service>.<namespace>.svc.cluster.local
• <pod-ip>.<namespace>.pod.cluster.local

#CNI Plugins

#Container Runtimes & Sandboxing (gVisor, Kata, Firecracker)

When a pod runs, something actually starts the container. That stack has layers, and the security-critical question is how strong the isolation is between the container and the host kernel.

   kubelet
     │  speaks CRI (Container Runtime Interface)
     ▼
   containerd  or  CRI-O          ← the "high-level" runtime (manages images, lifecycle)
     │  calls a "low-level" runtime
     ▼
   runc (default)  │  gVisor (runsc)  │  Kata Containers  │  Firecracker (via Kata)
   ───────────────────────────────────────────────────────────────────────────────
   shares host     │  user-space      │  real lightweight │  microVM (~125ms boot,
   kernel directly │  kernel intercepts│ VM per pod        │  minimal device model)
   (fast, weakest  │  syscalls (strong │ (own kernel,      │  (AWS Lambda/Fargate
    isolation)     │  isolation, some  │  strong isolation,│  use this under the hood)
                   │  perf cost)       │  more overhead)   │

Memory hook

these exist because "a container is a process, not a VM." Default runc containers share the host kernel, so one kernel-level container-escape bug compromises every workload on the node. gVisor, Kata, and Firecracker each put a barrier back between the container and the host kernel — gVisor by emulating the kernel in user space (intercepting syscalls), Kata/Firecracker by wrapping the pod in a genuine lightweight VM. The spectrum is isolation vs speed: runc is fastest/weakest, microVMs are strongest/heaviest. You reach for them when running untrusted or multi-tenant code — e.g., a CI system or a platform that runs customers' arbitrary containers. Interviewers love "how would you safely run untrusted containers?" → answer: gVisor or Kata/Firecracker, not plain runc.

#Service Mesh & Envoy

As soon as you have many services talking to each other, you need consistent encryption, retries, traffic routing, and observability between them — without baking it into every app. That's a service mesh, and its workhorse is Envoy.

Envoy is a high-performance L7 proxy. In a mesh it runs as a sidecar — a second container injected into every pod — so all traffic in and out of the app goes through its local Envoy:

   ┌──────── Pod A ────────┐         ┌──────── Pod B ────────┐
   │  app  ⇄  Envoy sidecar│ ──mTLS──│ Envoy sidecar  ⇄  app │
   └───────────────────────┘         └───────────────────────┘
         (app talks plaintext to its local Envoy; the Envoys
          encrypt + authenticate the wire between pods)

   Control plane (Istio's istiod / Linkerd) configures all the Envoys:
     who can talk to whom, mTLS certs, routing rules, retries, telemetry

Memory hook

a service mesh is "TLS + authz + telemetry for east-west traffic, moved out of the app into a sidecar." The app keeps speaking plain HTTP to localhost; the injected Envoy proxy transparently handles mTLS encryption, who-can-call-whom authorization, retries, and metrics for it. Security value: you get encryption-in-transit and identity-based access control between every microservice without trusting developers to implement it, plus per-call visibility that's gold for detection. The cost is operational complexity and a sidecar in every pod. Mnemonic: mesh = mutual-TLS + metrics + routing, as infrastructure.

#Storage & Persisting Data

Containers are ephemeral — when a pod dies, its writable layer is gone. So "where does the data live?" is a real design question.

   Ephemeral (dies with the pod)        Persistent (survives pod restart/reschedule)
   ──────────────────────────────       ─────────────────────────────────────────────
   • container writable layer           • PersistentVolume (PV)  ← real storage (EBS, GCE PD, NFS)
   • emptyDir volume                    • PersistentVolumeClaim (PVC) ← a pod's request for a PV
   • ConfigMap / Secret (read-only)     • StorageClass ← dynamically provisions PVs on demand
                                        • bound to a StatefulSet for stable identity (databases)

Memory hook

"PVC is the request, PV is the storage, StorageClass is the vending machine." A pod doesn't grab disk directly; it files a claim (PVC: "give me 20Gi, fast SSD"), and either an admin pre-made a matching PV or a StorageClass dynamically provisions one from the cloud (an EBS volume on EKS, a PD on GKE) via a CSI driver. Stateful things (databases) use a StatefulSet so each replica keeps a stable name and its own persistent volume across restarts. The reality-check most people learn: for serious databases, many teams use the cloud's managed DB (RDS/Cloud SQL) instead of running stateful DBs in K8s at all — K8s shines for stateless services. Security angle: a PV often outlives pods, so it can hold sensitive data and snapshots that need encryption and access control of their own.

#Scheduling

# Node affinity
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: [us-east-1a]

# Resource requests and limits
resources:
  requests:
    cpu: "250m"
    memory: "64Mi"
  limits:
    cpu: "500m"
    memory: "128Mi"

#ConfigMaps and Secrets

# Secret (base64, not encrypted at rest unless KMS envelope encryption enabled)
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  password: cGFzc3dvcmQ=   # base64("password") — NOT secure without etcd encryption

# Consume as env var
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: password

Secure secret storageenable etcd encryption at rest (AES-GCM KMS provider) or use External Secrets Operator + cloud secret manager.

#Probes

livenessProbe:    # restart container if fails
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

readinessProbe:   # remove from Service endpoints if fails
  httpGet:
    path: /ready
    port: 8080

startupProbe:     # delay other probes until app is started
  httpGet:
    path: /started
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

#How Developers Authenticate to the Cluster

A crucial real-world point: Kubernetes has no built-in user database. There are no "user accounts" in etcd for humans. The API server authenticates requests by trusting an external mechanism, then authorizes with RBAC. So "how do people log in" = "what external identity does the API server trust?"

   kubectl  ──(credential in kubeconfig)──▶  kube-apiserver
                                               │ 1. AUTHENTICATE (who are you?)
                                               │     • client cert (rare for humans)
                                               │     • OIDC token (Okta/Entra/Google) ← modern
                                               │     • cloud IAM (EKS/GKE) ← managed clusters
                                               │ 2. AUTHORIZE (RBAC: what can you do?)
                                               │ 3. ADMISSION (policy checks)
                                               ▼
                                              etcd

#The common human-auth options

#Wiring Okta (or any IdP) to a cluster — the shape of it

   1. Register the cluster as an OIDC app in Okta (issuer URL, client ID)
   2. Point the kube-apiserver at Okta:
        --oidc-issuer-url=https://company.okta.com
        --oidc-client-id=kubernetes
        --oidc-username-claim=email
        --oidc-groups-claim=groups        ← Okta groups → K8s RBAC groups
   3. Bind Okta groups to RBAC:
        a RoleBinding gives group "okta:platform-admins" the cluster-admin ClusterRole
   4. Developers run `kubectl` via a helper (kubelogin/Dex) that does the Okta SSO
      flow, gets a short-lived OIDC token, and kubectl presents it.

Memory hook

K8s authenticates by trust delegation, not its own user list. There's no kubectl create user. The API server says "I trust Okta (or AWS IAM, or this CA) to tell me who you are," then RBAC decides what that identity can do. So adding a team is really "map your IdP groups to RBAC RoleBindings." Why it matters for security: you get SSO, MFA, and instant offboarding for free (disable the person in Okta → cluster access gone), and the tokens are short-lived so there's no standing kubeconfig secret to steal. The classic anti-pattern is sharing one long-lived admin kubeconfig — no attribution, no revocation.

#Service accounts vs human users

#Local vs EKS vs GKE — What's Actually Different

The big distinction is who runs the control plane (API server, etcd, scheduler). That determines what you secure and what's the cloud's problem.

Memory hook

managed clusters move the control plane (and its risk) to the cloud; you still own the workloads. On EKS/GKE you never see etcd or the API server hosts — the cloud patches and secures them, which removes a huge class of "is my control plane hardened?" worries (and the etcd-exposure incidents that plagued self-managed clusters). What stays yours on every platform: RBAC, network policy, pod security, secrets, and the images you run. The other big difference is identity glue — EKS and GKE bolt the cloud's IAM onto cluster auth (IAM → RBAC) and onto pod-to-cloud access (IRSA / Workload Identity), so you manage one identity system instead of two. GKE Autopilot goes furthest — you don't manage nodes at all.

#Monitoring, Image Scanning & "Is Something Wrong?"

Three different questions, three different toolsets.

1. Is the cluster healthy / behaving normally? (observability)

   Metrics  → Prometheus (scrapes pods/nodes) + Grafana (dashboards) + Alertmanager
   Logs     → Fluent Bit/Fluentd → Loki / Elasticsearch / cloud logging
   Traces   → OpenTelemetry → Jaeger/Tempo (request flow across services)
   "Golden signals": latency, traffic, errors, saturation

2. Is the image safe? (image scanning — shift left + at runtime)

3. Is the configuration wrong / drifting? (posture + admission + runtime)

Memory hook

scan the image before it runs, enforce policy as it's admitted, watch behavior while it runs. Three time-points: build/registry (image CVE scanning + signing — "shift left"), admission (policy gate — reject bad configs and unsigned images before they're scheduled), and runtime (Falco/eBPF — catch what got through). A mature setup has all three, because each catches what the others miss: scanning misses logic/config flaws, admission misses zero-days, runtime is the last line. "How do you know if something's wrong with an image or config?" → name those three layers.

#Key kubectl Commands

# Context / cluster
kubectl config get-contexts
kubectl config use-context <ctx>

# Inspect
kubectl get pods -n <ns> -o wide
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
kubectl exec -it <pod> -n <ns> -- /bin/sh

# RBAC
kubectl auth can-i create pods --as=system:serviceaccount:default:my-sa
kubectl get rolebindings,clusterrolebindings --all-namespaces | grep <subject>

# Audit
kubectl get events --sort-by=.lastTimestamp -n <ns>

#Reality Check: Do You Actually Dump Memory in Container IR?

You asked the right question — and the honest answer is rarely, in most teams. Classic host DFIR puts memory acquisition near the top (order of volatility). Container/cloud-native IR usually works differently, for practical reasons:

Why memory dumping is uncommon in container IR:

When teams do reach for memory:

• Fileless / in-memory malware that leaves little on disk (a crypto-miner injected into a process, an in-memory implant) — here RAM is where the evidence lives.
• Advanced/targeted intrusions where you need to recover injected code, keys, or decrypted configs.
• Malware reverse-engineering, not routine SOC triage.
• In practice it's usually a node-level capture (snapshot the node's disk and/or memory) or grabbing /proc/<pid>/ of the suspect process, done by a specialized DFIR function — not the everyday IR responder.

Memory hook

The honest interview answer"In container IR I prioritize the API audit log, runtime/eBPF telemetry, logs, and the image — they answer 'what happened' faster and more reliably than pod memory, and the workload is ephemeral and gets rebuilt anyway. I'd only do memory forensics for fileless/in-memory malware or a targeted intrusion, and even then it's typically a node-level capture by a DFIR specialist. So no — routine pod memory dumping is not standard practice, and saying otherwise would be cargo-culting host forensics into a cloud-native world." Saying this — that you match the technique to the environment — signals real experience.