[3/3] Complete guide to CI/CD pipelines with Drone.io on kubernetes — Drone metrics with Prometheus

4 min readApr 5, 2020

original article https://blog.cogarius.com/index.php/2020/04/05/complete-guide-to-ci-cd-pipelines-with-drone-io-on-kubernetes-3-3-drone-metrics-with-prometheus/

TL;DR;

You are running kubernetes and using an expensive yet easy and maintainable CI/CD pipelines.

You want to save money but don’t want to spend too much time migrating and don’t want to give up on features.

You want to be able to :

Push your images to your private docker registry
Monitor your build with prometheus
Access your hashicorp vault secrets from your pipeline.

This series of three articles will help you go through it with Drone CI !

Check the first post to setup the private registry, the drone server and the kube runner. The second post is about vault integration with drone to retrieve your secret in your pipelines. In this post we will detail the Prometheus and Grafana configuration in order to scrape and display drone metrics.

Prometheus

We are running Prometheus operator. As stated in the documentation, the Prometheus Operator introduces additional resources in Kubernetes to declare the desired state of a Prometheus and Alertmanager cluster as well as the Prometheus configuration. The resources it introduces are:

Prometheus
Alertmanager
ServiceMonitor

The Prometheus resource declaratively describes the desired state of a Prometheus deployment, while a ServiceMonitor describes the set of targets to be monitored by Prometheus.

Therefore we simply need a ServiceMonitor to scrape the drone metrics. The only drawback is that the drone metrics endpoint is restricted and requires an authorization token.

If you go to navigate to your prometheus UI and see on the Status/Targets page something like below. It probably means that you need to authenticate to the drone metrics endpoint.

$kubectl port-forward -n monitoring \
prometheus-op-prometheus-operator-prometheus-0 8284:9090

Drone configuration

A Drone account is needed to access the Drone metrics endpoint. We will need to Drone CLI to achieve this. To find your token simply go to the drone UI, click on your profile in the right upper corner and then user settings.

Basically you need two environment variables for the CLI to work

$export DRONE_SERVER=https://drone.mycompany.com
$export DRONE_TOKEN=3xc56f4d5s6f564sd54f8ds67fs

Let’s create a user for Prometheus

$ drone user add prometheus --machine 
Successfully added user prometheus
Generated account token e5a68798d7f8787fd0b3d4918d46

Kubernetes configuration

Let’s create a kubernetes secret, on the same namespace we deployed Prometheus, based on this token. We will then mount it into the Prometheus deployment.

$ kubectl create secret generic drone-metrics\
 --from-literal=token=e5a68798d7f8787fd0b3d4918d46

We can now add the secret into the Prometheus operator values chart

prometheusSpec:  
    ## Secrets is a list of Secrets in the same namespace as the Prometheus object, which shall be mounted into the Prometheus Pods.
    ## The Secrets are mounted into /etc/prometheus/secrets/. Secrets changes after initial creation of a Prometheus object are not
    ## reflected in the running Pods. To change the secrets mounted into the Prometheus Pods, the object must be deleted and recreated
    ## with the new list of secrets.
    ##
    secrets: 
      - drone-metrics

Note that the Secrets are mounted into /etc/prometheus/secrets/. Let’s upgrade the chart.

$ helm upgrade op stable/prometheus-operator -f values.yaml

Prometheus has now a token to authenticate to the drone’s metrics endpoint.

Service monitor

We can now create a ServiceMonitor resource to indicate where and how Prometheus can scrape drone’s metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: prometheus-operator
    release: op
  name: drone
  namespace: drone
spec:
  endpoints:
  - bearerTokenFile: /etc/prometheus/secrets/drone-metrics/token
    port: http
  selector:
    matchLabels:
      app.kubernetes.io/component: server
      app.kubernetes.io/name: drone

You can notice that we provided the token through the bearerTokenFile field. You can take a look at the all the endpoints fields available.

After the ServiceMonitor is created you should be able to check again in the targets page of the Prometheus UI. The newly created drone target should be up.

How does Prometheus know which ServiceMonitor to use?

We added special labelrelease:op to the ServiceMonitor. Indeed Prometheus Operator will select service monitor based on its config.

$ kubectl get -n monitoring  prometheus \
  op-prometheus-operator-prometheus -o json |\
  jq '.spec.serviceMonitorSelector.matchLabels'
{
  "release": "op"
}

This means that in my setup for Prometheus Operator to take into consideration ServiceMonitor resources they must have the label release:op. thanks to managedkube.com for the tip

Grafana dashboard

With the metrics available in Prometheus all we have to do is to display them with elegance and style. Hopefully grafana dashboard can help us do exactly that !

You can find my version of the Drone Grafana dashboard here. We display the active and total builds alongside the CPU, network and memory usage of the current build jobs.

That’s all folks! I hope you will enjoy Drone has much as we do. If you have questions remarks you can PM me: telegram:@Zgorizzo mail: ben@cogarius.com

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.