IBM WebSphere Application Server Performance Cookbook

OpenShift Investigate ImagePullBackOff Recipe

ImagePullBackOff occurs when an image can't be pulled. This may have many causes such as insufficient authorization, pulling from the context of one project but the ImageStream is in another project, network issues or protections, etc.

  1. Ensure you're logged in with oc
  2. Run the command that causes the issue and note the date and time (in UTC). For example: 
    $ TZ=UTC date
Tue Feb 16 18:36:10 UTC 2021
$ oc debug node/$NODE -t --image=image-registry.openshift-image-registry.svc:5000/$PROJECT/$IMAGE        
Creating debug namespace/openshift-debug-node-9jmxs ...
Starting pod/...-debug ...
To use host binaries, run `chroot /host`

Removing debug pod ...
Removing debug namespace/openshift-debug-node-9jmxs ...
error: Back-off pulling image "image-registry.openshift-image-registry.svc:5000/$PROJECT/$IMAGE"
  3. Review the logs of the image registry at approximately the date and time of the issue. For example: 
    $ oc logs deployments/image-registry -n openshift-image-registry | grep -v -e probe -e metrics
[...]
time="2021-02-16T18:36:18.148593608Z" level=warning msg="error authorizing context: authorization header required" go.version=go1.13.15 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=17430d0b-49ec-4ba9-8161-32456ca2f29f http.request.method=GET http.request.remoteaddr="...:50939" http.request.uri=/v2/ http.request.useragent="cri-o/1.18.3-19.rhaos4.5.git9264b4f.el7 go/go1.13.15 os/linux arch/amd64"
  4. In the above example, the image pull back-off occurred because of an authorization issue. In this case, it is because we forgot to specify -n $PROJECT on the oc debug command which was run in the context of another project.