Share via

Deploy a collectd agent in a Kubernetes cluster on Azure

  • Article

Authored by Stas Kondratiev. Edited by Nanette Ray. Reviewed by Ben Hummerstone.

To gather metrics about your Kubernetes cluster in Azure, you can collect telemetry from the nodes and store it in Graphite, InfluxDb, or another time-series database. This article describes how to use collectd, a popular daemon that collects system and application metrics and can store them in many ways.

Collectd gained popularity for its performance and light weight (it’s written in C). With more than 90 plugins, collectd is a flexible tool for system monitoring including containers.

The collectd agent runs in a Docker image, meaning you run collectd in a completely containerized environment. By writing the telemetry to a Graphite-enabled database, you can then monitor metrics and events in your cluster.

Prerequisites

This article assumes that you have:

  • A Graphite-enabled server to receive and aggregate the collectd metrics. It’s beyond the scope of this article to describe how to deploy this type of server, which takes careful planning. One approach is to use a Linux virtual machine on Azure.
  • A Kubernetes cluster deployed in Azure using Azure Container Service.
  • A private Azure Container Registry to store your custom Docker images.
  • A local installation of Docker, used to create a custom Docker image.

Create a collectd Docker image

For maximum flexibility and portability, you can create a Docker image to run your collectd agent. These setup steps are based on Alpine Linux, a lightweight Linux distribution, but the Docker image can be created on either Windows or Linux as the local operating system.

  1. Create a new folder named collectd-agent, then change to that directory by using the command:
 cd collectd-agent
  1. In this folder, create a file named Dockerfile and add the following text to it:
 # A simple collectd agent on Linux Alpine

FROM alpine:3.5



RUN apk update && apk add collectd



CMD [ "/bin/sh", "-c", "collectd; while :; do sleep 300; done" ]

 

This file describes a Docker image based on Alpine Linux that installs collectd and sets it up as an entry point.

  1. To build a Docker image based on this Dockerfile, run the following command in shell. For <your registry name>, substitute the name of the Azure Container Registry you previously created.
 docker build -t <your registry name>-microsoft.azurecr.io/collectd-agent:1.0 .
  1. To push the Docker image to the Azure Container Registry, run the following command, substituting the name of your Azure Container Registry for <your registry name>:
 docker push <your registry name>-microsoft.azurecr.io/collectd-agent:1.0

NOTE: You may have to log on to Azure Container Registry if you haven’t before. You will be prompted for your logon credentials.

Deploy to Kubernetes cluster

To deploy the collectd agent to the Kubernetes cluster, YAML files are used to specify the configuration details. You create a service that provides the IP address of the Graphite-enabled server and then deploy it to the Kubernetes cluster.

You also define a ConfigMap, a YAML file that loads the plugins used by collectd. For this example, we load the following three plugins, but you can add more as needed:

  • CPU plugin. Collects the amount of time spent by a CPU in various states, such as executing user code, executing system code, waiting for I/O operations, and being idle.
  • Memory plugin. Collects physical memory usage by the operating system, such as used, buffered, cached, and free.
  • Write Graphite plugin. Stores values in Carbon, the storage layer of a Graphite installation.

To run the monitoring agent, you create DaemonSets. Kubernetes uses DaemonSets to run a single instance of a container on each host in the cluster.

  1. Create a file named graphite-svc.yml that contains the following content specifying a Kubernetes service and an external endpoint for a Graphite server. This is a convenient way to decouple the external installation of a Graphite server from your Kubernetes cluster. For the ip parameter, specify the IP address of your Graphite server.
 # graphite service

apiVersion: v1

kind: Service

metadata:

  name: graphite-svc

spec:

  clusterIP: 10.0.90.1 # can be any cluster IP you want

  ports:

  - port: 2003

    targetPort: 2003

    protocol: TCP

---

apiVersion: v1

kind: Endpoints

metadata:

  name: graphite-svc

subsets:

  - addresses:

    - ip: <your graphite server IP address>

    ports:

    - port: 2003

      protocol: TCP
  1. To deploy the service and endpoint to your Kubernetes cluster, run the following command:
 kubectl apply -f graphite-svc.yml
  1. Create a ConfigMap file named collectd-config.yml with the following content specifying the collectd configuration and plugins:
 kind: ConfigMap

apiVersion: v1

metadata:

  name: collectd-config

  namespace: default

data:

  node-collectd.conf: |-

    FQDNLookup false

    LoadPlugin syslog

    <Plugin syslog>

       LogLevel info

    </Plugin>




    LoadPlugin cpu

    LoadPlugin memory

    LoadPlugin write_graphite




    <Plugin "cpu">

      Interval 5

      ReportByState false

      ReportByCpu false

    </Plugin>




    <Plugin "memory">

      Interval 30

      ValuesAbsolute false

      ValuesPercentage true

    </Plugin>




    <Plugin write_graphite>

       <Node "wgnode1">

          Host "10.0.90.1"

          Port "2003"

          Protocol "tcp"

          LogSendErrors true

          StoreRates true

          AlwaysAppendDS false

          EscapeCharacter "_"

       </Node>

    </Plugin>

    #Last line (collectd requires ‘\n’ at the last line)

 

  1. If you didn’t do so earlier, create a secret for your Azure Container Registry using the following command:
 kubectl create secret docker-registry acrregkey --docker-server=<your registry name>-microsoft.azurecr.io --docker-username=<name> --docker-password=<password> --docker-email=<your email>

To generate the password for your container registry, use Azure Portal.

  1. Create a Kubernetes DaemonSet file named collectd-node-agent.yml that contains the following content describing the collectd agent. For the image parameter, specify your registry name.
 apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: collectd-node-agent

  labels:

    app: collectd-node-agent

spec:

  template:

    metadata:

      labels:

        name: collectd-node-agent

    spec:

      hostNetwork: true

      imagePullSecrets:

        - name: acrregkey

      containers:

      - name: collectd

        image: <your registry name>-microsoft.azurecr.io/collectd-agent:1.0

        securityContext:

          privileged: true

        volumeMounts:

        - name: collectd-config

          mountPath: /etc/collectd

        - name: proc

          mountPath: /mnt/proc

          readOnly: true

        - name: root

          mountPath: /hostfs

          readOnly: true

        - name: etc

          mountPath: /mnt/etc

          readOnly: true

        - name: run

          mountPath: /var/run/docker.sock

      volumes:

      - name: collectd-config

        configMap:

          name: collectd-config

          items:

          - key: node-collectd.conf

            path: collectd.conf

      - name: proc

        hostPath:

          path: /proc

      - name: root

        hostPath:

          path: /

      - name: etc

        hostPath:

          path: /etc

      - name: run

        hostPath:

          path: /var/run/docker.sock

 

  1. To deploy the DaemonSet to your Kubernetes cluster, run the following command:
 kubectl apply -f collectd-node-agent.yml

The collectd agent starts to collect telemetry from the nodes in your Kubernetes cluster and writes the data to your Graphite-enabled server.

Next steps

The following resources provide helpful tips:

Comments