Everything You Need to Know to Setup & Use Kube State Metrics

⇩ Download!

Priyansh Khodiyar

Author

Share with friends

Kube State Metrics, often abbreviated as KSM, is like the backstage pass to the inner workings of your Kubernetes cluster.

Imagine you're running a big show with multiple actors (containers) on a stage (nodes), and you want to know what's happening behind the curtain.

helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

KSM is your behind-the-scenes guide, providing essential information about the state of your cluster.

In simple terms, KSM collects data about everything in your Kubernetes cluster, from running pods and services to nodes and resource usage.

It then turns this data into metrics that you can use to monitor, troubleshoot, and optimize your cluster's performance.

Whether you're a Kubernetes newbie or a seasoned pro, understanding Kube State Metrics is like having a magic map to navigate the Kubernetes universe with confidence.

Let's dive in and uncover the secrets it holds!

What is Kube State Metrics?

Kube State Metrics is an open-source project in the Kubernetes ecosystem that focuses on providing insights into the state of resources in a Kubernetes cluster.

It collects and exposes various metrics related to the objects and components within a Kubernetes cluster, making it easier for administrators and operators to monitor the health and performance of their clusters.

Key features of Kube State Metrics

Metrics Collection

Kube State Metrics collects metrics by querying the Kubernetes API server and converting the resource information into a structured format that can be easily consumed by monitoring and observability tools.

Resource Types

It provides metrics for a wide range of Kubernetes resources, including pods, nodes, namespaces, services, replication controllers, and many others. This allows you to track the state and performance of these resources over time.

Custom Metrics

In addition to the built-in resource metrics, Kube State Metrics can also expose custom metrics. This is useful for capturing specific information about your applications and services running in Kubernetes.

Also Read: Understanding eBPF in Kubernetes

Prometheus Integration

Kube State Metrics is commonly used in conjunction with Prometheus, a popular open-source monitoring and alerting system.

It exposes metrics in a format that Prometheus can scrape, making it a valuable data source for Kubernetes monitoring.

Visualization and Alerting

Once the metrics are collected and made available to tools like Prometheus, you can use Grafana or other visualization and alerting solutions to create dashboards and set up alerts based on the collected data.

Also Read: Datadog vs Grafana

How to Setup Kube-State-Metrics?

Step 1: Create a Kubernetes Namespace

You can create a dedicated Kubernetes namespace for Kube State Metrics to isolate it from other resources in your cluster.

To create a namespace, you can use the following command:

kubectl create namespace kube-state-metrics

Step 2: Deploy Kube State Metrics

Here's a sample YAML deployment file for Kube State Metrics:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-state-metrics # If you created a namespace in Step 1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v2.2.0 # Use the latest version
        ports:
        - containerPort: 8181

Save this YAML configuration to a file, such as kube-state-metrics-deployment.yaml, and then apply it to your cluster:

kubectl apply -f kube-state-metrics-deployment.yaml

This will deploy Kube State Metrics as a single replica within your cluster.

Also Read: Guide to Kubernetes ReplicaSets

Step 3: Accessing Kube State Metrics

KubeState metrics exposes its metrics on port 8181 by default.

To access the metrics, you can use port forwarding to the local port:

kubectl port-forward -n kube-state-metrics svc/kube-state-metrics 8181:8181

Now, you can access the metrics locally at [http://localhost:8181/metrics](http://localhost:8181/metrics).

Step 4: Integrating with Prometheus

To make Kube State Metrics data available to Prometheus, you need to configure Prometheus to scrape the Kube State Metrics endpoint.

Here's an example Prometheus configuration snippet:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: 'kube-state-metrics'
        static_configs:
          - targets: ['kube-state-metrics:8181']

Apply this configuration to your Prometheus instance:

kubectl create configmap prometheus-config -n <your-prometheus-namespace> --from-file=prometheus.yml=<path-to-prometheus-config.yaml>

Don't forget to adjust the job_name and targets as needed.

Step 5: Restart Prometheus

If Prometheus is already running, you may need to restart it to pick up the new configuration.

You can do this by deleting and recreating the Prometheus pods or using a rolling restart depending on your Prometheus deployment strategy.

After completing these steps, Prometheus will start scraping metrics from Kube State Metrics, and you can use Prometheus and Grafana to create dashboards, and alerts, and gain insights into the state of your Kubernetes resources.

Deploy Kube State Metrics Using Helm

To deploy Kube State Metrics using Helm, you can use a community-contributed chart.

Here's a general guideline on how to do it.

Step 1: Add Helm Repository

If you haven't already added the Helm repository where the Kube State Metrics chart is located, you can do so using the helm repo add command.

For example:

helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

Step 2: Install Kube State Metrics Helm Chart

You can then install the Kube State Metrics Helm chart using the helm install command.

Here's an example command:

helm install kube-state-metrics kube-state-metrics/kube-state-metrics

This command will deploy Kube State Metrics in your cluster using the default chart values.

Also Read: What are Custom Resource Definitions (CRDs) in Kubernetes?

Step 3: Customize Configuration

You can customize the deployment by specifying your own values.yaml file or by using the \--set flag with helm install_ to override specific configuration options.

For example:

helm install kube-state-metrics kube-state-metrics/kube-state-metrics \
   --set rbac.create=true \
   --set service.enabled=true

This command sets RBAC (Role-Based Access Control) to true and enables the Kubernetes service associated with Kube State Metrics.

Step 4: Verify Deployment

You can check the status of the deployment to ensure it was successful:

helm list

This will show you a list of installed Helm releases, including the one for Kube State Metrics.

Also Read: Kubectl Cheat Sheet

How does Kube-State-Metrics Collect Data?

Kube State Metrics collects data from a Kubernetes cluster by interacting with the Kubernetes API server. It queries the API server to retrieve information about the state and configuration of various Kubernetes objects and components.

Here's how Kube State Metrics collects data.

HTTP Requests to Kubernetes API Server

Kube State Metrics communicates with the Kubernetes API server using HTTP requests. It sends requests to specific API endpoints to retrieve information about the cluster's resources.

Resource Types

Kube State Metrics is designed to collect data about various types of Kubernetes resources, including but not limited to:

• Pods

• Nodes

• Namespaces

• Deployments

• ReplicaSets

• Services

• ConfigMaps

• Secrets

• StatefulSets

• CronJobs

• Jobs

• Ingresses

• Persistent Volumes

• Persistent Volume Claims

API Queries

For each resource type, Kube State Metrics queries the relevant API endpoint to fetch information about those resources.

For example, to collect data about pods, it queries the /api/v1/pods endpoint.

Data Conversion

Once the data is retrieved from the API server, Kube State Metrics processes and converts it into a structured format.

This format typically adheres to the Prometheus exposition format, which is widely used in the Kubernetes monitoring ecosystem.

Metrics Endpoint

Kube State Metrics exposes this processed data as metrics via an HTTP endpoint.

By default, it serves these metrics on port 8181. This endpoint allows external monitoring and observability tools, such as Prometheus, to scrape the collected metrics.

Also Read: Top NGNIX Ingress Configuration Options

Scraping by Monitoring Tools

Monitoring and observability tools like Prometheus can be configured to scrape the metrics exposed by Kube State Metrics at regular intervals.

Prometheus, for example, can be configured with target URLs pointing to the Kube State Metrics endpoint.

Visualization and Alerting

Once the metrics are scraped and ingested by the monitoring tool, they can be visualized on dashboards, used for alerting, and integrated with various monitoring and alerting solutions.

This allows administrators and operators to gain insights into the state of the Kubernetes cluster and its resources.

How to Use Kube State Metrics?

The use cases of kube-state-metrics are spread across your level of implementing it. Let's look at different ways of using Kube State Metrics and their respective use cases.

Step 1: Deploy Kube State Metrics

Deploy Kube State Metrics in your Kubernetes cluster. You can use a Helm chart or apply a YAML deployment as previously explained.

a. Use Case 1: Resource State Monitoring

KSM collects data about various Kubernetes resources like pods, nodes, namespaces, services, deployments, and more.

b. Use Case 2: Resource Configuration Changes

Monitor changes in resource configurations, including labels, annotations, and other metadata. This helps you track and audit configuration drift.

Also Read: Understanding Configuration as Code

Step 2: Configure Prometheus

Configure Prometheus to scrape metrics from Kube State Metrics.

scrape_configs:
  - job_name: 'kube-state-metrics'
    static_configs:
      - targets: ['kube-state-metrics:8181']

In this example, you have defined a Prometheus job named 'kube-state-metrics' and specified the target URL where Prometheus can scrape Kube State Metrics.

c. Use Case 3: Resource Metrics Collection

Collect metrics about resource counts in your cluster.

For example, monitor the number of pods per namespace or the number of nodes.

d. Use Case 4: Resource Label Filtering

Utilize Prometheus to filter and group resources based on labels or annotations.

For instance, monitor resources labeled with specific environment or application names.

Step 3: Deploy Grafana

Deploy Grafana in your cluster to create dashboards and visualize metrics collected by Prometheus.

e. Use Case 5: Dashboard Creation

Design custom Grafana dashboards to visualize Kube State Metrics data. Create dashboards for pod resource utilization, node health, or service response times.

f. Use Case 6: Cluster Health Monitoring

Monitor the overall health of your Kubernetes cluster by visualizing metrics such as node status, pod status, and resource utilization on Grafana dashboards.

Step 5: Monitor and Maintain

• Regularly monitor and maintain your Kube State Metrics, Prometheus, and Grafana deployments:

• Ensure Kube State Metrics stays up-to-date.

• Monitor Prometheus and Grafana's performance.

• Periodically review and update alerting rules and dashboards as your cluster and application requirements evolve.

Kube State Metrics Best Practices

Here are the best practices for using Kube State Metrics:

• Keep KSM up to date.

• Secure KSM metrics endpoint.

• Implement RBAC for access control.

• Use consistent resource labeling.

• Customize metrics collection.

• Monitor and set up alerts in Prometheus.

• Create informative Grafana dashboards.

• Maintain documentation for your monitoring setup.

• Monitor the monitoring stack itself.

• Plan for resource usage.

• Regularly review and update configurations.

• Implement backup and disaster recovery.

• Consider metrics retention policies.

• Test and validate alerting rules and dashboards.

• Leverage community resources for support and knowledge sharing.

Also Read: How to Use Kubectl Logs?

Kube-State-Metrics vs Metrics Server

Kube-State-Metrics and Metrics Server are two distinct components used for monitoring and collecting metrics in Kubernetes clusters. They serve different purposes and have different capabilities.

Kube-State-Metrics

• Purpose: Kube-State-Metrics is primarily used for collecting and exposing metrics about the state and configuration of Kubernetes resources, such as pods, nodes, namespaces, services, and more. It provides a detailed snapshot of the current state of these resources.

• Metrics Types: Kube-State-Metrics collects metrics that are related to the static configuration and state of resources. Examples include the number of pods in each phase (Running, Pending, Failed), resource labels, annotations, and other metadata.

• Data Source: Kube-State-Metrics collects data directly from the Kubernetes API server by making HTTP queries to specific API endpoints. It does not collect runtime or performance metrics for containers.

• Use Cases: Kube-State-Metrics is suitable for creating dashboards and alerts that focus on resource configurations, health, and static state. It's often used in combination with monitoring tools like Prometheus and Grafana for cluster introspection and observability.

Metrics Server

• Purpose: Metrics Server, also known as kubelet-instrumentation or kubelet-metrics-server, is designed to provide resource utilization metrics for pods and nodes in a Kubernetes cluster. It focuses on collecting metrics about CPU and memory usage, network usage, and other performance-related data.

• Metrics Types: Metrics Server primarily collects metrics related to resource utilization, including CPU and memory usage per pod and node. These metrics are essential for monitoring and scaling workloads based on resource consumption.

• Data Source: Metrics Server collects data directly from the kubelets running on individual nodes in the cluster. It queries kubelets to obtain real-time performance metrics.

• Use Cases: The Metrics Server is essential for Horizontal Pod Autoscaling (HPA) and Cluster Autoscaler functionality in Kubernetes. It's used for making scaling decisions based on resource usage, ensuring efficient resource allocation, and maintaining cluster health.

Also Read: What is an Internal Developer Platform - IDP?

Kube State Metrics vs. Node Exporter

Kube State Metrics and Node Exporter are two distinct monitoring components often used in Kubernetes environments, each serving different purposes.

Kube State Metrics

• Purpose: Kube State Metrics is designed specifically for collecting and exposing metrics related to the state and configuration of Kubernetes resources. It focuses on providing insights into the Kubernetes objects and their static characteristics.

• Metrics Types: Kube State Metrics collects metrics related to Kubernetes resources such as pods, nodes, namespaces, services, deployments, and more. These metrics include information like pod counts, resource labels, and annotations.

• Data Source: Kube State Metrics queries the Kubernetes API server to gather data about the cluster's state. It does not collect system-level metrics from the underlying nodes.

• Use Cases: Kube State Metrics is mainly used to understand the state of Kubernetes resources and create dashboards and alerts related to Kubernetes object configurations and statuses. It's often integrated with monitoring tools like Prometheus for this purpose.

Node Exporter

• Purpose: Node Exporter, on the other hand, is part of the Prometheus ecosystem and is used to collect system-level metrics from the underlying nodes in a Kubernetes cluster. It focuses on monitoring the health and performance of individual nodes.

• Metrics Types: Node Exporter gathers metrics related to the host machine, including CPU usage, memory usage, disk I/O, network traffic, and more. These metrics provide insights into the node's resource utilization and health.

• Data Source: Node Exporter collects metrics directly from the host's operating system and hardware, rather than from Kubernetes APIs. It runs as a separate process on each node in the cluster.

• Use Cases: Node Exporter is essential for monitoring the overall health and performance of the nodes in a Kubernetes cluster. It helps identify resource bottlenecks, hardware failures, or performance issues that could impact the stability and performance of the cluster.

With this, you have reached the end of the blog and it's time for a quick conclusion.

Summary- Kube-State-Metrics

To summarize it all, Kube State Metrics is a vital tool for monitoring Kubernetes environments.

This blog covered its definition, and setup with a practical example including Helm charts, data collection methods, and usage scenarios with Prometheus and Grafana integration.

You've explored some best practices for its effective deployment and compared it with Metrics Server and Node Exporter.

With this comprehensive understanding, you can harness Kube State Metrics to gain deep insights, optimize cluster health, and enhance the observability of your Kubernetes infrastructure, making informed decisions and ensuring efficient resource utilization.

Share with friends

Written by Priyansh Khodiyar

TwitterLinkedIn

Priyansh is the founder of UnYAML and a software engineer with a passion for writing. He has good experience with writing and working around DevOps tools and technologies, APMs, Kubernetes APIs, etc and loves to share his knowledge with others.

Further Reading