[3/3] Complete guide to CI/CD pipelines with Drone.io on kubernetes — Drone metrics with Prometheus
You are running kubernetes and using an expensive yet easy and maintainable CI/CD pipelines.
You want to save money but don’t want to spend too much time migrating and don’t want to give up on features.
You want to be able to :
- Push your images to your private docker registry
- Monitor your build with prometheus
- Access your hashicorp vault secrets from your pipeline.
This series of three articles will help you go through it with Drone CI !
Check the first post to setup the private registry, the drone server and the kube runner. The second post is about vault integration with drone to retrieve your secret in your pipelines. In this post we will detail the Prometheus and Grafana configuration in order to scrape and display drone metrics.
We are running Prometheus operator. As stated in the documentation, the Prometheus Operator introduces additional resources in Kubernetes to declare the desired state of a Prometheus and Alertmanager cluster as well as the Prometheus configuration. The resources it introduces are:
The Prometheus resource declaratively describes the desired state of a Prometheus deployment, while a
ServiceMonitor describes the set of targets to be monitored by Prometheus.
Therefore we simply need a
ServiceMonitor to scrape the drone metrics. The only drawback is that the drone metrics endpoint is restricted and requires an authorization token.
If you go to navigate to your prometheus UI and see on the Status/Targets page something like below. It probably means that you need to authenticate to the drone metrics endpoint.
$kubectl port-forward -n monitoring \
A Drone account is needed to access the Drone metrics endpoint. We will need to Drone CLI to achieve this. To find your token simply go to the drone UI, click on your profile in the right upper corner and then
Basically you need two environment variables for the CLI to work
Let’s create a user for Prometheus
$ drone user add prometheus --machine
Successfully added user prometheus
Generated account token e5a68798d7f8787fd0b3d4918d46
Let’s create a kubernetes secret, on the same namespace we deployed Prometheus, based on this token. We will then mount it into the Prometheus deployment.
$ kubectl create secret generic drone-metrics\
We can now add the secret into the Prometheus operator values chart
## Secrets is a list of Secrets in the same namespace as the Prometheus object, which shall be mounted into the Prometheus Pods.
## The Secrets are mounted into /etc/prometheus/secrets/. Secrets changes after initial creation of a Prometheus object are not
## reflected in the running Pods. To change the secrets mounted into the Prometheus Pods, the object must be deleted and recreated
## with the new list of secrets.
Note that the Secrets are mounted into /etc/prometheus/secrets/. Let’s upgrade the chart.
$ helm upgrade op stable/prometheus-operator -f values.yaml
Prometheus has now a token to authenticate to the drone’s metrics endpoint.
We can now create a
ServiceMonitor resource to indicate where and how Prometheus can scrape drone’s metrics.
- bearerTokenFile: /etc/prometheus/secrets/drone-metrics/token
You can notice that we provided the token through the
bearerTokenFile field. You can take a look at the all the endpoints fields available.
ServiceMonitor is created you should be able to check again in the targets page of the Prometheus UI. The newly created drone target should be up.
How does Prometheus know which ServiceMonitor to use?
We added special label
release:op to the
ServiceMonitor. Indeed Prometheus Operator will select service monitor based on its config.
$ kubectl get -n monitoring prometheus \
op-prometheus-operator-prometheus -o json |\
This means that in my setup for Prometheus Operator to take into consideration
ServiceMonitor resources they must have the label
release:op. thanks to managedkube.com for the tip
With the metrics available in Prometheus all we have to do is to display them with elegance and style. Hopefully grafana dashboard can help us do exactly that !
You can find my version of the Drone Grafana dashboard here. We display the active and total builds alongside the CPU, network and memory usage of the current build jobs.
That’s all folks! I hope you will enjoy Drone has much as we do. If you have questions remarks you can PM me: telegram:@Zgorizzo mail: email@example.com