Use Case 2. Pod Deletion Process - Internals

📎 Reference: https://lnkd.in/gaqp7-Xg

See what happens internally when you run kubectl delete pod, and why Graceful Shutdown settings matter.

1. Overall Flow

    sequenceDiagram
    participant U as kubectl
    participant A as API Server
    participant E as etcd
    participant K as kubelet
    participant EP as EndpointSlice Controller
    participant P as kube-proxy
    U->>A: DELETE pod
    A->>E: Set deletionTimestamp (Terminating)
    par Block traffic
        EP->>EP: Remove Pod from EndpointSlice
        EP-->>P: Propagate endpoint change
    and Terminate container
        K->>K: Run preStop hook
        K->>K: Send SIGTERM
        K->>K: Wait for terminationGracePeriodSeconds
        K->>K: SIGKILL if not terminated
    end
    K->>A: Report Pod termination
    A->>E: Final deletion of Pod object

Delete request: The API server does not remove the Pod immediately; it sets metadata.deletionTimestamp (status: Terminating).
Endpoint removal: The EndpointSlice controller removes the Pod from the Service’s backend list.
kubelet termination handling: preStop hook → SIGTERM → wait for the grace period → SIGKILL.
Final deletion: When the kubelet reports termination, the API server deletes the object from etcd.

⚠️ Steps 2 and 3 proceed in parallel. This means SIGTERM can arrive before endpoint propagation has finished, and some requests may flow into the terminating Pod in the meantime.

2. Important Settings

spec:
  terminationGracePeriodSeconds: 30   # Default is 30 seconds
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 5"]   # Buys time for endpoint propagation

Setting	Role
`terminationGracePeriodSeconds`	The time allowed between SIGTERM and SIGKILL
`preStop` hook	Runs before SIGTERM. Used for connection draining, a short sleep, etc.
`kubectl delete --grace-period=0 --force`	Removes the Pod object from the API immediately, without a grace period (containers may remain on the node, so this is dangerous)

3. When a Pod Is Stuck in Terminating

Cause	Check/Action
The app ignores SIGTERM (PID 1 problem)	Run the app directly (or use `exec` in the shell entrypoint) and implement a SIGTERM handler
A Finalizer remains	`kubectl get pod <p> -o jsonpath='{.metadata.finalizers}'`
The node is `NotReady`	The kubelet is unresponsive → recover the node or force-delete the Pod
Volume detach delay	Check the CSI driver logs

kubectl delete pod <name> --grace-period=0 --force   # Last resort

4. Summary

Deletion = set deletionTimestamp → remove from EndpointSlice + (preStop → SIGTERM → SIGKILL) → delete from etcd. If you want zero-downtime deployments, be sure to design for a preStop sleep and SIGTERM handling.

Use Case 1. How Kubernetes Enforces Resource Quota Use Case 3. Resolving a Kubernetes Node in NotReady State