[3/3] Complete guide to CI/CD pipelines with Drone.io on kubernetes — Drone metrics with Prometheus
original article https://blog.cogarius.com/index.php/2020/04/05/complete-guide-to-ci-cd-pipelines-with-drone-io-on-kubernetes-3-3-drone-metrics-with-prometheus/
TL;DR;
You are running kubernetes and using an expensive yet easy and maintainable CI/CD pipelines.
You want to save money but don’t want to spend too much time migrating and don’t want to give up on features.
You want to be able to :
- Push your images to your private docker registry
- Monitor your build with prometheus
- Access your hashicorp vault secrets from your pipeline.
This series of three articles will help you go through it with Drone CI !
Check the first post to setup the private registry, the drone server and the kube runner. The second post is about vault integration with drone to retrieve your secret in your pipelines. In this post we will detail the Prometheus and Grafana configuration in order to scrape and display drone metrics.
Prometheus
We are running Prometheus operator. As stated in the documentation, the Prometheus Operator introduces additional resources in Kubernetes to declare the desired state of a Prometheus and Alertmanager cluster as well as the Prometheus configuration. The resources it introduces are:
Prometheus
Alertmanager
ServiceMonitor
The Prometheus resource declaratively describes the desired state of a Prometheus deployment, while a ServiceMonitor
describes the set of targets to be monitored by Prometheus.
Therefore we simply need a ServiceMonitor
to scrape the drone metrics. The only drawback is that the drone metrics endpoint is restricted and requires an authorization token.
If you go to navigate to your prometheus UI and see on the Status/Targets page something like below. It probably means that you need to authenticate to the drone metrics endpoint.
$kubectl port-forward -n monitoring \
prometheus-op-prometheus-operator-prometheus-0 8284:9090
Drone configuration
A Drone account is needed to access the Drone metrics endpoint. We will need to Drone CLI to achieve this. To find your token simply go to the drone UI, click on your profile in the right upper corner and then user settings
.
Basically you need two environment variables for the CLI to work
$export DRONE_SERVER=https://drone.mycompany.com
$export DRONE_TOKEN=3xc56f4d5s6f564sd54f8ds67fs
Let’s create a user for Prometheus
$ drone user add prometheus --machine
Successfully added user prometheus
Generated account token e5a68798d7f8787fd0b3d4918d46
Kubernetes configuration
Let’s create a kubernetes secret, on the same namespace we deployed Prometheus, based on this token. We will then mount it into the Prometheus deployment.
$ kubectl create secret generic drone-metrics\
--from-literal=token=e5a68798d7f8787fd0b3d4918d46
We can now add the secret into the Prometheus operator values chart
prometheusSpec:
## Secrets is a list of Secrets in the same namespace as the Prometheus object, which shall be mounted into the Prometheus Pods.
## The Secrets are mounted into /etc/prometheus/secrets/. Secrets changes after initial creation of a Prometheus object are not
## reflected in the running Pods. To change the secrets mounted into the Prometheus Pods, the object must be deleted and recreated
## with the new list of secrets.
##
secrets:
- drone-metrics
Note that the Secrets are mounted into /etc/prometheus/secrets/. Let’s upgrade the chart.
$ helm upgrade op stable/prometheus-operator -f values.yaml
Prometheus has now a token to authenticate to the drone’s metrics endpoint.
Service monitor
We can now create a ServiceMonitor
resource to indicate where and how Prometheus can scrape drone’s metrics.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: prometheus-operator
release: op
name: drone
namespace: drone
spec:
endpoints:
- bearerTokenFile: /etc/prometheus/secrets/drone-metrics/token
port: http
selector:
matchLabels:
app.kubernetes.io/component: server
app.kubernetes.io/name: drone
You can notice that we provided the token through the bearerTokenFile
field. You can take a look at the all the endpoints fields available.
After the ServiceMonitor
is created you should be able to check again in the targets page of the Prometheus UI. The newly created drone target should be up.
How does Prometheus know which ServiceMonitor to use?
We added special labelrelease:op
to the ServiceMonitor
. Indeed Prometheus Operator will select service monitor based on its config.
$ kubectl get -n monitoring prometheus \
op-prometheus-operator-prometheus -o json |\
jq '.spec.serviceMonitorSelector.matchLabels'
{
"release": "op"
}
This means that in my setup for Prometheus Operator to take into consideration ServiceMonitor
resources they must have the label release:op
. thanks to managedkube.com for the tip
Grafana dashboard
With the metrics available in Prometheus all we have to do is to display them with elegance and style. Hopefully grafana dashboard can help us do exactly that !
You can find my version of the Drone Grafana dashboard here. We display the active and total builds alongside the CPU, network and memory usage of the current build jobs.
That’s all folks! I hope you will enjoy Drone has much as we do. If you have questions remarks you can PM me: telegram:@Zgorizzo mail: ben@cogarius.com