Use Case 3. Resolving a Kubernetes Node in NotReady State

📎 Reference: https://lnkd.in/gdH9SVuN

A troubleshooting procedure that systematically narrows down why a Node goes into the NotReady state.

1. What Is NotReady?

A Node is NotReady when its kubelet fails to report its status (Lease/NodeStatus) to the API server periodically, or when the reported Ready condition is False or Unknown. After a grace period (40 seconds by default), the node lifecycle controller marks the Node as NotReady and later evicts the Pods on that Node.

    flowchart TD
    A["NotReady Node found"] --> B{"kubectl describe node<br/>Check Conditions"}
    B -->|Unknown: Kubelet stopped posting| C["Check kubelet / Node connectivity"]
    B -->|MemoryPressure / DiskPressure / PIDPressure| D["Relieve resource pressure"]
    B -->|NetworkUnavailable| E["Check CNI"]
    C --> F{systemctl status kubelet}
    F -->|inactive| G["Restart kubelet / check configuration"]
    F -->|active| H["Check container runtime, certificates, API server connectivity"]
  

2. Step-by-Step Checks

Step 1. Check the Conditions

kubectl get nodes
kubectl describe node <node> | sed -n '/Conditions/,/Addresses/p'
kubectl get events --field-selector involvedObject.name=<node>

Step 2. Log in to the Node and check the kubelet

systemctl status kubelet
journalctl -u kubelet -n 100 --no-pager

Step 3. Check the container runtime

systemctl status containerd
crictl ps
crictl info

Step 4. Check for resource pressure

df -h        # DiskPressure (/var/lib/kubelet, /var/lib/containerd)
free -m      # MemoryPressure

Step 5. Network, CNI, and certificates

ls /etc/cni/net.d/
kubectl get pods -n kube-system -o wide | grep <node>
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates
curl -k https://<apiserver>:6443/healthz

3. Common Causes

4. Verification After Recovery

kubectl uncordon <node>      # Uncordon the Node if you had cordoned it
kubectl get nodes -w

If the Node cannot be recovered, run kubectl drain <node> --ignore-daemonsets --delete-emptydir-data and then replace the Node.