A Complete Guide to Kubernetes Events & Kubectl Get Events

June 5, 2024

English

Priyansh Khodiyar

Share with friends

In this guide, we will closely look at Kubernetes Events and the intricacies of the kubectl get events command. We will talk about everything from the types of Kubernetes events to how to use kubectl get events with real-world examples.

A Complete Guide to Kubernetes Events & Kubectl Get Events cover image

Applications don't always run in a steady state, as you might expect. User activity varies depending on the day, the hour, or certain occasions. Runtime needs for memory, networking, and CPU fluctuate.

It's crucial to react to these occurrences effectively. However, it's as crucial to comprehend how and why particular apps are behaving in this way.

Let's investigate how Kubernetes event objects are used to provide some context.

What are Kubectl Events - The Basics

In Kubernetes, Kubectl Events provide eventful insights into the state of your cluster, helping you monitor and troubleshoot its health and performance.

Think of them as real-time notifications that you get on your phone, here, they inform you about various occurrences within your Kubernetes environment.

These events are emitted by Kubernetes components, including controllers, schedulers, and other system-level entities.

Kubectl Events covers a wide range of activities, such as pod creations, deletions, errors, node issues, and more.

They act as an invaluable source of information for developers to understand what's happening under the hood and to identify potential problems before they escalate within a certain time frame.

Yes, these events do not stay for eternity but for one hour post the occurrence of the event.

By default configuration, events are stored in etcd for a limited amount of time, say 60 minutes, and are made accessible by kubectl commands such as kubectl get events or kubectl describe pod.

Also Read: What are Kubernetes Operators?

Where do Kubernetes Events Get Stored?

Kubernetes events are API objects that get stored in the apiserver on master in your cluster.

Now that you know what Kubernetes events are, it will make more sense to understand their various types.

Also Read: Docker vs Containerd

Types of Kubectl Events - Kubernetes Event Classifications

Kubernetes events are a helpful tool for monitoring your system and catching potential problems. By keeping track of these events and setting up alerts, you can quickly detect and resolve issues.

These events can be broadly classified into five types. Familiarizing yourself with these five types of events will help you understand event tracking better.

1. Failed Events

These kubectl events occur when there's an error with your object manifest file or a problem pulling a container image from the repository.

To troubleshoot this, check the image address and tags using the command: docker pull <image_name>.

Common causes point towards incorrect credentials for private repositories, rate limiting, or possible typos in the image name or tags.

Also Read: How to Setup & Manage Kubernetes Cluster Using Kubeadm?

2. Evicted Events

If a node in your Kubernetes cluster lacks sufficient resources, which is not that rare, it may evict pods with resource constraints.

But, you can prevent evictions by using taints to stop scheduling new pods on heavily utilized nodes.

3. Failed Scheduling Events

These events happen when there aren't enough nodes available to schedule a new pod. Kinda a cranky situation!

To diagnose the issue, use: kubectl describe object/object-name.

Also Read: Kubernetes Node vs Pod vs Cluster

4. Volume Events

As we know persistent volumes are crucial for data storage in Kubernetes workloads, if you encounter "FailedMount" or "FailedAttachVolume" events, it could be due to networking or configuration errors between persistent volumes and claims.

To debug, use kubectl describe pv and kubectl describe pvc to get more details about the volumes and claims.

5. Node Events

Nodes are the machines in your cluster running applications and storing data but, unhealthy nodes can lead to various issues, like 5XX errors or problems with deployments.

To troubleshoot this, you can try rebooting the node manually or use the control plane to do it automatically.

Let's look at some other types of events/states in Kubernetes.

1. CrashLoopBackOff - This takes place whenever a pod starts, crashes, restarts, and crashes once more.

2. ImagePullBackOff - When the node is unable to obtain the picture.

3. Evicted Events - These can occur when a node decides to terminate or evict a pod in order to free up resources (CPU, memory, etc.). K8s is supposed to reschedule the pod on another node if this occurs.

4. FailedMount / FailedAttachVolume - this event stops pods from starting if the persistent volume or storage they need cannot be accessed.

5. FailedSchedulingEvents - When the scheduler is unable to locate a node on which to run your pods.

6. NodeNotReady - when a node is unable to launch a pod due to a more fundamental problem.

How to Use Kubectl Get Events (with Examples)

The kubectl get events command is your window into the world of Kubernetes events. You can use it to fetch valuable insights into your cluster's activities.

Here are some examples of using kubectl get events

1. Displaying All Events

To view all events occurring within your Kubernetes cluster, use the following command:

kubectl get events

2. Filtering Events by Namespace

If you want to focus on events from a specific namespace, use the --namespace flag:

kubectl get events --namespace mynamespace

3. Watching Events in Real-time

For real-time monitoring of events, you can use the watch flag with the kubectl events command:

kubectl get events --watch

4. Getting Events for a Specific Object

To retrieve events related to a specific object, such as a pod or deployment, use the object's name and specify its type:

kubectl get events --field-selector involvedObject.name=my-pod --field-selector involvedObject.kind=Pod

5. Filtering Events by Type

As mentioned earlier, you can filter events based on their types (Normal, Warning, or Error) using the --field-selector flag:

kubectl get events --field-selector type=Warning

Also Read: A Complete Guide to Kubectl Proxy

How to Access Kubernetes Events?

By default, Kubernetes events are stored in the cluster and can be accessed using the following command:

kubectl get events

Explanation of the output columns:

LAST SEEN: The time elapsed since the Kubernetes event was last seen or recorded.
TYPE: The type of the event, which can be Normal, Warning, or Error.
REASON: The reason for the event, indicating the nature of the event.
OBJECT: The object associated with the event, such as a pod or a deployment.
MESSAGE: A short description or message providing more details about the event.

In the sample output:

A pod named nginx-deployment-abc123 was successfully scheduled on node-1.
The pod is currently in the process of pulling the latest nginx image from the repository.
The image was pulled successfully in approximately 1.23 seconds, and the container nginx was created.
The nginx container in the pod has started successfully.
There's a warning indicating that the nginx-deployment-abc123 pod is experiencing back-off restarting due to a failed container.
Another warning event indicates that the pod nginx-deployment-xyz456 failed to schedule because no nodes were available that tolerate its taints.

If you are not using one of the popular monitoring tools, you can use kubectl or access the API directly using a script. Simply pass the -o json flag to any obtain operation to get JSON output from kubectl.

kubectl get events -o json

Also Read: How to Run Multiple Apps Using Just One LoadBalancer in Kubernetes?

How to Filter & Monitor Kubectl Events?

Our kubectl get events command can produce a lot of information, especially in big clusters. To make it more manageable and to make logical output, we can filter events using the --field-selector flag.

For instance, if we only want to see events related to pods, we can use this command:

kubectl get events --field-selector involvedObject.kind=Pod -n <namespace>

What this will do is it will display events where the involvedObject is a pod. You can also apply other selectors to further narrow down events based on criteria like reason, type, and source.

You can create your own exporter to export Kubernetes events if Prometheus is configured to monitor your cluster. When events occur, you can then set up notifications in Prometheus using AlertManager.

By setting the DD_COLLECT_KUBERNETES_EVENTS environment variable for your agents to true, you can enable Kubernetes event collection if you use Datadog.

After that, you can set up monitors for the event-related purposes of your choice.

Moreover, the eventhandler_config is a Grafana Agent Kubernetes integration that lets your agent retrieve real-time events from the Kubernetes API and transmit them to Grafana Loki, which serves as your log aggregator.

How to Filter Kubernetes Events Based on Timestamps?

Use kubectl get events --sort-by='.metadata.creationTimestamp' -A to sort according to the creation time of each namespace.

To sort out the occurrence of events by “last seen” timestamp, use the below command:

kubectl get events --sort-by='.lastTimestamp'

Frequently Asked Questions

1. How can I see what's happening in Kubernetes?

To see what's happening in Kubernetes, you can use the command kubectl get events. It will show you a list of recent activities in your cluster.

2. How can I view logs for events in a Kubernetes pod?

To view logs for events in a Kubernetes pod, you can use the command kubectl describe pod. It will give you detailed information about the pod and any related events.

3. What's the command to see events for a specific pod?

To see events for a specific pod, use the command kubectl get events --field-selector involvedObject.name=my-pod --field-selector involvedObject.kind=Pod. Just replace my-pod with the name of your pod.

4. How can I keep an eye on Kubernetes events?

To keep an eye on Kubernetes events in real-time, use the command kubectl get events --watch. It will continuously show you new events as they happen in your cluster.

5. How do I check events for the whole Kubernetes cluster?

To check events for the entire Kubernetes cluster, use the command kubectl get events. By default, it shows events from the entire cluster, giving you an overview of its activities. If you want to focus on specific namespaces or objects, you can add more options to the command using the --namespace flag as shown earlier.