Use Case 4. Kubernetes ImagePullBackOff Explained

📎 Reference: https://lnkd.in/gW8Q9c29

Why a Pod ends up in the ErrImagePull → ImagePullBackOff state, and how to fix it.

1. What the States Mean

    flowchart LR
    A[Pod scheduled] --> B[kubelet attempts image pull]
    B -->|Success| C[Container starts]
    B -->|Failure| D[ErrImagePull]
    D --> E[Waiting in ImagePullBackOff]
    E -->|Retry| B
  

2. Diagnosis Steps

kubectl get pod <pod>
kubectl describe pod <pod>      # The Failed / Back-off messages in Events are the key

Events example:

Failed to pull image "myapp:v2": rpc error: code = NotFound ... not found
Failed to pull image "registry.example.com/app": ... pull access denied, authorization failed

3. Fixes by Cause

Configuring imagePullSecrets

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=<user> \
  --docker-password=<password> \
  -n <namespace>

spec:
  imagePullSecrets:
  - name: regcred
  containers:
  - name: app
    image: registry.example.com/team/app:1.0.0

4. Checking Directly on the Node

crictl pull registry.example.com/team/app:1.0.0

Because the pull is performed by the node (kubelet/containerd) rather than the Pod, reproducing it on the node is the fastest way.

5. Procedure for Using Multi-Architecture Images

The error no matching manifest for linux/arm64/v8 in the manifest list entries occurs when the image was not built for the node’s CPU architecture (amd64/arm64). A multi-architecture image is a manifest list (OCI index) that bundles images for several architectures under a single tag, and the runtime automatically selects the image matching the node’s architecture.

    flowchart LR
    T["myapp:1.0 (manifest list)"] --> A["linux/amd64 image"]
    T --> B["linux/arm64 image"]
    N1[amd64 node] -->|pull| A
    N2[arm64 node] -->|pull| B
  

Step 1. Check the Node Architecture

kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture

Step 2. Check Which Architectures the Image Supports

docker buildx imagetools inspect registry.example.com/team/app:1.0.0
# or
docker manifest inspect registry.example.com/team/app:1.0.0 | grep -A3 platform

Both Platform: linux/amd64 and linux/arm64 must be present.

Step 3. Build and Push a Multi-Architecture Image (buildx)

# First time only: create a multi-platform builder
docker buildx create --name multiarch --use
docker buildx inspect --bootstrap

# Build and push in one step (multi-platform images are not loaded locally; they are pushed directly to the registry)
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.example.com/team/app:1.0.0 \
  --push .

• Building with QEMU emulation is slow, so in CI consider using native runners per architecture or cross-compilation in the Dockerfile (--platform=$BUILDPLATFORM, TARGETARCH).
• The base image must also support the target architectures.

Step 4. Merge Manifests When Per-Architecture Images Were Built Separately

docker buildx imagetools create \
  -t registry.example.com/team/app:1.0.0 \
  registry.example.com/team/app:1.0.0-amd64 \
  registry.example.com/team/app:1.0.0-arm64

Step 5. Deploy to the Cluster

Specify a single image tag, and each node pulls the image matching its own architecture.

spec:
  containers:
  - name: app
    image: registry.example.com/team/app:1.0.0   # manifest list tag

For a single-architecture image that must be deployed only to nodes of a specific architecture, restrict scheduling with a nodeSelector.

spec:
  nodeSelector:
    kubernetes.io/arch: arm64

Step 6. Verify

kubectl get pods -o wide                       # Confirm Running on each node
crictl pull --platform linux/arm64 registry.example.com/team/app:1.0.0   # Reproduce directly on the node

6. Prevention Tips

• Use immutable version tags or a digest (@sha256:...)
• Understand how imagePullPolicy works: IfNotPresent / Always / Never
• The registry Secret must exist in every Namespace that pulls from the private registry (you can attach imagePullSecrets to a ServiceAccount)