A Complete Kubernetes imagePullPolicy Tutorial & Troubleshooting

June 5, 2024

English

Priyansh Khodiyar

Share with friends

Kubernetes is used as the most widely used orchestrator tool. The applications are deployed using containerized images. What's the process behind this?

A Complete Kubernetes imagePullPolicy Tutorial & Troubleshooting cover image

In this tutorial, we are going to discuss all about managing container registries and images and how to work with them in Kubernetes.

Container Images

A containerized image is your application that is packaged with only necessary libraries on a lightweight base image pulled from multiple container registries available in the market such as the Docker registry.

Updating Container Images

Every time you make a change in your application code, you need to build the image again and push it to the container registry. This procedure needs proper attention as you will be creating multiple versions of your application.

So if you choose to maintain every version of your application, you can opt for different tags for your image such as <image>:v1, <image>:v2, etc.

These images created on different tags can be pushed to your container registry and used further for your deployments.

Docker commands are used to push the images onto the Docker registry or any cloud provider's registry.

Enough theory, let's look at some commands.

Use this command to build the containerized image of your application.

docker build -t <image_name>:<tag><path_where_dockerfile_exist>

To tag the image to your container registry, use the following command.

docker tag <image_name> <registry_login_server_name>/<image_name>:<tag>

To push the image

docker push <registry_login_server_name>/<image_name>:<tag>

The image stored in these registries needs to be pulled using a manifest file.

Here's an example of a pod file showcasing the basic configurations to deploy your container images onto Kubernetes.

apiVersion: v1
kind: Pod
metadata:
  name: container-pod
spec:
  containers:
    - name: container-image-name
      image: container-image:latest
      imagePullPolicy: IfNotPresent

Here, the keyword “image” is used to get the image contained in the container registry.

This will be in the format: <login_server_name>/<image_name>:<tag>. The next configuration is mentioned as imagePullPolicy.

This is the concept we are going to elaborate on and talk about everything you need to know about it.

Also Read: Docker Image vs Docker Container

What is imagePullPolicy in Kubernetes?

Kubernetes imagePullPolicy controls when and how Kubernetes has to fetch container images for cluster-running pods.

It gives you control over how the container runtime retrieves images from a registry. The imagePullPolicy is defined in the pod specification and can be configured at both the pod and deployment levels.

Also Read: Top 13 Docker Alternatives

Types of imagePullPolicy in Kubernetes

IfNotPresent imagePullPolicy

This policy instructs Kubernetes to pull the image only if it is not already present on the node.

If the image already exists locally, Kubernetes will use it instead of attempting to pull it from the registry again. This policy is beneficial if your network connection is slow or limited and you want to avoid unwanted pulls.

When you wish to share updated versions of your image using the same tag, forced pulls come in handy.

This could happen if you tag photographs with the branch name from which they were built. If the tag was already accessible locally, Kubernetes would never pull your new image releases if the Always policy was not in place.

Also Read: The Only Tutorial on Kubernetes DaemonSet That You'll Ever Need

Always imagePullPolicy

This policy specifies that the container image specified in the manifest file has to be pulled every time the file is run using the kubelet command.

This is the default setting when you don't mention this configuration in the file.

This policy is suitable for frequently updated images or when you want to ensure that the latest version is used every time we run the manifest file to deploy pods.

However, keep in mind that it may cause unnecessary image pulls if the image hasn't changed.

Also, it works best only in scenarios where you are using certain specific tags such as the latest tag for a container image.

Also Read: Kubernetes StatefulSet with Best Practices & Examples

Never imagePullPolicy

This policy is the exact opposite of the “Always” imagePullPolicy. It is used to tell Kubernetes to never pull the container image mentioned.

This policy has to be used in scenarios where we are certain that the image is already present on the node. Suppose the image is not already present on the node, the pod fails to start.

This imagePullPolicy is useful when you want to avoid accidental image pulls or when you're using a custom process to ensure images are preloaded on nodes.

In the example shown in the previous example, the imagePullPolicy can be set in the spec section of a pod's definition file, either at the pod level or at the container level.

The imagePullPolicy for the container "container-pod" is set to "IfNotPresent," indicating that Kubernetes should only pull the image if it is not already present on the node.

The kubelet command does not directly set the image pull policy. It is specified within the pod/deployment definition file. Kubelet takes care of pulling container images, starting and stopping containers, and monitoring containers' health.

Kubernetes will never modify imagePullPolicy as a consequence of another action.

Editing a Pod's image will not trigger Kubernetes to re-evaluate the default pull policy.

That means that if you start with my-image:latest but later update the Pod to my-image:my-release, the imagePullPolicy will still be IfNotPresent.

You should manually specify a new policy if one is desired.

How Do You Check a Container's imagePullPolicy Configuration?

To check the image pull policy settings for a container, use kubectl CLI commands to inspect the pod configuration.

Here are a few commands you can use to retrieve the image pull policy details.

To check the image pull policy for containers in a specific pod:

kubectl get pod <pod-name> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.imagePullPolicy}{"\n"}{end}'

Replace <pod-name> with the name of the pod that you want to inspect.

Here's how you can check the image pull policy for containers in a specific deployment:

kubectl get deployment <deployment-name> -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.imagePullPolicy}{"\n"}{end}'

Replace <deployment-name> with the name of the deployment that you want to inspect.

To check the image pull policy for containers in all pods:

kubectl get pods --all-namespaces -o jsonpath='{range .items[*].spec.containers[*]}{.name}{"\t"}{.imagePullPolicy}{"\n"}{end}'

This command will display the container name along with its image pull policy for all pods in a JSON format and for all the namespaces in Kubernetes.

The output will show the container name and its corresponding image pull policy (e.g., "Always," "IfNotPresent," or "Never") for each container within the specified pod/deployment.

Also Read: Best Practices for Kubernetes Namespaces, Security, Resource Limits

All about imagePullSecrets

imagePullSecrets is also another container image configuration available for a pod/deployment. It is used to specify the authentication and authorization details of your container registry which helps you to log in and pull container images in a registry.

This setting is particularly useful when you are using a private registry.

Here's how ImagePullSecrets work.

Create a Kubernetes (K8s) Secret

You need to create a K8s secret containing the necessary authentication credentials of the registry like a username and password or a token.

The secret can be created manually or using a Kubernetes Secret manifest file. The details added while creating a secret have to be base64encoded.

Associate the Secret with the Pod

Once the secret is created, associate it with the deployment/pod that needs to pull the private image.

Here is where the imagePullSecrets field to the pod's specification is added and the secret created in the previous step is referenced.

Pulling Private Container Images

Now when you apply the manifest file, the credentials from the secret referenced are taken and login is attempted on the node.

If authentication is successful, the container image is pulled and deployed onto Kubernetes as a deployment or a pod.

The manifest file for the above procedure is done as shown below.

apiVersion: v1
kind: Secret
metadata:
  name: container-secret
data:
  username: hfoefk==
  password: fghoanfpiauewhfpQWNT[]iruy49900==
---
apiVersion: v1
kind: Pod
metadata:
  name: container-pod
spec:
  containers:
    - name: container-image
      image: private-registry/container-image:latest
  imagePullSecrets:
    - name: container-secret

Notice that the secret manifest with the name “container-secret” is created first which holds the registries' username and password. The same secret is referenced in the pod manifest as highlighted.

Common Issues in Container Image References

imagePullBackOff

imagePullBackOff is one of the most common errors in the pod that occurs commonly due to authentication issues or incorrect references to an image.

For instance, if you provided the wrong container registry name or wrong image name, the status of the pod post-deployment will be imagePullBackOff.

To check all pods in all namespaces that have a status of "imagePullBackOff", execute the following command:

kubectl get pods --all-namespaces --field-selector status.phase=Failed --field-selector status.reason=ImagePullBackOff

Here's how to troubleshoot 'imagePullBackOff'.

Verify image: Check if the image reference is correct. Ensure that the image name, tag, and registry information are accurate.
Check availability of the image: Ensure that the image exists in the specified container registry and is accessible. Verify the image availability and permissions for the Kubernetes cluster.
Verify image pull secrets: If the image is from a private registry, ensure that the username/password information is accurate.
Check network connectivity: Verify that the cluster nodes are connected to networks so that it is able to reach the container registry. Ensure that any required firewall rules, proxies, or network configurations are set.

ErrImagePull

Another common issue we face in the pods with respect to container images is ErrImagePull.

This is a straightforward error that suggests that the container image reference provided is unable to pull.

This can also occur due to the image not being in the same network, wrong image name, etc.

Here's how you can find out the exact issue in your pod.

kubectl describe pod <pod_name> -n <namespace>

This gives you all the details of the pod and the events that occurred in it from the time of creation. This information will be crucial to troubleshooting issues related to container images.