A Vault for all your Secrets (full TLS on kubernetes with kv v2)
original article https://blog.cogarius.com/index.php/2020/03/13/a-vault-for-all-your-secrets-full-tls-on-kubernetes-with-kv-v2/
TL;DR
Install Vault via a Helm chart and configure it to access it through HTTPS all the way.
We will create a key value store (v2) and enable the userpass, approle and kubernetes authentication. We will define some policies and test them.
Finally we will configure the vault agent injector with TLS certificate to inject secrets through injector webhook thanks to annotations inside a POD.
Vault is a powerful tool for securing access to secrets. It comes with a lot of features like a Web UI, dynamic secrets, HA, data encryption, authentication, temporary secrets & revocation and logging.
TLS and certificates…for Vault
We will start by generating the certificates needed to enable TLS on vault inside the cluster. We will follow this guide for the certificates creation and we detail the helm chart further here below.
..for Vault agent injector certificates
We will also generate certificate for the vault injector in the same way. Here are the condensed steps to follow.
# SERVICE is the name of the Vault agent injector service
SERVICE=vault-agent-injector-svc# NAMESPACE where the Vault service is running.
NAMESPACE=vault# SECRET_NAME to create in the Kubernetes secrets store.
SECRET_NAME=vault-agent-injector-tls# TMPDIR is a temporary working directory.
TMPDIR=/tmp# Generate key for the certificate
$ openssl genrsa -out ${TMPDIR}/vault-injector.key 2048# Create a file ${TMPDIR}/csr-vault-agent-injector.conf
# with the following contents:$ cat <<EOF >${TMPDIR}/csr-vault-agent-injector.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE}
DNS.2 = ${SERVICE}.${NAMESPACE}
DNS.3 = ${SERVICE}.${NAMESPACE}.svc
DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
IP.1 = 127.0.0.1
EOF# Create a Certificate Signing Request (CSR).
$ openssl req -new -key ${TMPDIR}/vault-injector.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server-vault-agent-injector.csr -config ${TMPDIR}/csr-vault-agent-injector.conf$ export CSR_NAME=vault-agent-injector-csr
$ cat <<EOF >${TMPDIR}/agent-injector-csr.yaml
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: ${CSR_NAME}
spec:
groups:
- system:authenticated
request: $(cat ${TMPDIR}/server-vault-agent-injector.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF# Send the CSR to Kubernetes.
$ kubectl create -f ${TMPDIR}/agent-injector-csr.yaml# Approve the CSR in Kubernetes
$ kubectl certificate approve ${CSR_NAME}# Store key, cert, and Kubernetes CA into Kubernetes secrets store
# Retrieve the certificate.
$ serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')# Write the certificate out to a file.
$ echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault-injector.crt# Retrieve Kubernetes CA.
$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode > ${TMPDIR}/vault-injector.ca# for the helm chart we need to base64 encode the ca certificate
# to fill the value injector.certs.caBundle you can run
# cat ${TMPDIR}/vault-injector.ca | base64# Store the key, cert, and Kubernetes CA into Kubernetes secrets.
$ kubectl create secret generic ${SECRET_NAME} \
--namespace ${NAMESPACE} \
--from-file=vault-injector.key=${TMPDIR}/vault-injector.key \
--from-file=vault-injector.crt=${TMPDIR}/vault-injector.crt \
--from-file=vault-injector.ca=${TMPDIR}/vault-injector.ca
Vault Helm Chart
Let’s now clone the official Helm Chart for vault. Note that this chart needs Helm 3 and kubernetes 1.9+. I have successfully deployed the chart with helm v3.1.1 and kubernetes v1.17.3
$ git clone https://github.com/hashicorp/vault-helm.git
Inside the values.yaml file you need to set tlsDisable variable to false to enable TLS. Note that all the configuration of the chart is detailed here
global:
...
# TLS for end-to-end encrypted transport
tlsDisable: falseNow let’s configure the injector
injector:
# True if you want to enable vault agent injection.
enabled:trueNow let’s configure the certificates that we have generated
certs:
# secretName is the name of the secret that has the TLS certificate and
# private key to serve the injector webhook. If this is null, then the
# injector will default to its automatic management mode that will assign
# a service account to the injector to generate its own certificates.
secretName: vault-agent-injector-tls
# caBundle is a base64-encoded PEM-encoded certificate bundle for the
# CA that signed the TLS certificate that the webhook serves. This must
# be set if secretName is non-null.
# Here we have pasted the vault.ca encoded in base64
caBundle: "LS0tLS1CRUdJTiBDRVJUSU...(see here above)"
# certName and keyName are the names of the files within the secret for
# the TLS cert and private key, respectively. These have reasonable
# defaults but can be customized if necessary.
certName: vault-injector.crt
keyName: vault-injector.keyand for the server
server:
# extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
# used to include variables required for auto-unseal.
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
# GOOGLE_REGION: global
extraVolumes:
- type: secret
name: vault-server-tls # Matches the ssl certif generated
# Run Vault in "standalone" mode. This is the default mode that will deploy if
# no arguments are given to helm. This requires a PVC for data storage to use
# the "file" backend. This mode is not highly available and should not be scaled
# past a single replica.
standalone:
enabled: true
# config is a raw string of default configuration when using a Stateful
# deployment. Default is to use a PersistentVolumeClaim mounted at /vault/data
# and store data there. This is only used when using a Replica count of 1, and
# using a stateful set. This should be HCL.
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
storage "file" {
path = "/vault/data"
}Don’t forget to add some limitations on the resources
resources:
requests:
memory: 256Mi
cpu: 250
limits:
memory: 512Mi
cpu: 500m$ helm install ./vault-helm
Vault init and unsealing
at the end of the chart installation you will notice that the vault-0 pod will not switch to the ready state. Indeed we need to init and unseal the vault for it to be ready. For HA deployments, only one of the Vault pods needs to be initialized.
$ kubectl exec -ti vault-0 -- vault operator init
$ kubectl exec -ti vault-0 -- vault operator unseal
Vault will print out five unseal keys and a root token. Indeed vault secrets are encrypted with an encryption key that is itself encrypted with a master key. Vault does not store the master key. To decrypt the data, Vault must decrypt the encryption key which requires the master key.
Unsealing is the process of reconstructing this master key. Without at least 3 keys out of the five to reconstruct the master key, the vault will remain permanently sealed!
Web UI
We can now access the web UI. it isn’t exposed via a Service by default so you must use kubectl port-forward to visit the UI
$ kubectl port-forward vault-0 8200:8200
We can authenticate with the token method simply by pasting the root token. There are four main sections
- Secrets
Where all the secrets engine and secret path are located
- Access
Auth :List the enabled authentication method
Entities: Vault clients can be mapped as entities and their corresponding accounts with authentication providers can be mapped as aliases
Groups: Entities can be part of a group. Policies can be applied to a group
Leases: metadata containing information such as a time duration, renewability, and more. Vault promises that the data will be valid for the given duration, or Time To Live (TTL). Once the lease is expired, Vault can automatically revoke the data, and the consumer of the secret can no longer be certain that it is valid.
- Policies
Everything in Vault is path based, and policies are no exception. Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. This section discusses policy workflows and syntaxes.
Policies are deny by default, so an empty policy grants no permission in the system.
- Tools
Wrap/Unwrap/Rewrap
- Here you can wrap a token (or anything you like) in JSON format
- Data is encrypted. You can recover the data using the token returned by this page, but be careful because if you lose the token you won’t be able to retrieve your data!
- You can unwrap the data with the token
- You can rewrap your data to rotate the token
Lookup: You can see some basic information about your wrapped data, including the expiration time.
Random: Random bytes generator with different options
Hash: Different hashing algorithm
UserPass Auth
Let’s enable the UserPass (username /password) authentication. To do so you can either do it through the vault browser CLI (accessible through the icon in the upper right corner of the web UI) or by connecting to the vault POD
$ kubectl exec -n vault vault-0 /bin/sh -it
$ vault login
TOKEN(enter the root token):
$ vault auth enable userpass
$ vault write auth/userpass/users/jeanblaguin \
password=yolo \
policies=jokes-policyWe successfully enabled userpass auth and created a user jeanblaguin with a password yolo that is link to the policy jokes-policy
kv secret engine
We need now to store some secret. We will use the kv (key/value) version 2 secret engine. Kv secret engine is used to store arbitrary secrets within the configured physical storage for Vault
$ vault secrets enable kv-v2
in the web UI you should see in the secret tab the newly created kv engine. Let’s now add a secret.
$ vault kv put kv/secret/jokes/oneliner goodone="I dreamed I was forced to eat a giant marshmallow. When I woke up, my pillow was gone."
# access the secret
$ vault kv get kv/secret/jokes/oneliner
====== Metadata ======
Key Value
--- -----
created_time 2020-03-06T16:26:37.861335217Z
deletion_time n/a
destroyed false
version 1
===== Data =====
Key Value
--- -----
goodone I dreamed I was forced to eat a giant marshmallow. When I woke up, my pillow was gone.
$ vault kv get -field=goodone kv/secret/jokes/oneliner
I dreamed I was forced to eat a giant marshmallow. When I woke up, my pillow was gone.
# get the metadata for all the versions
$ vault kv metadata get kv/secret/jokes/oneliner
# delete the secret (don't we will need it afterwards)
$ vault kv metadata delete secret/my-secret
Success! Data deleted (if it existed) at: secret/metadata/my-secretlet’s now create the corresponding jokes policies that will allow our user to access that secret.
Note that version 2 kv store uses a prefixed API, which is different from the version 1 API. Because writing and reading versions are prefixed with the data/ we should take this into account while writing our policies. For instance even though in the Web UI navigating to our secret will show a path like kv/secret/jokes in the acl we will write kv/data/secret/jokes. To know more about that change read the doc here
Policy
Create a file named jokes-policy.hcl in the /tmp folder and paste the here below content.
Here is the full documentation about policies. Note that required_parameters allow_parameters and and denied_parameters are not working with kv v2.
# This section grants all access on "kv/data/secret/jokes/*". Further restrictions can be# applied to this broad policy, as shown below.path "kv/data/secret/jokes/*" {capabilities = ["create", "read", "update", "delete", "list"]}
# Even though we allowed secret/*, this line explicitly denies# kv/data/secret/jokes/super-joke. This takes precedence.path "kv/data/secret/jokes/super-joke" {capabilities = ["deny"]}
Let’s create the policy with the file
$vault policy write jokes-policy jokes-policy.hcl
Success! Uploaded policy: jokes-policy
You can read the policy
$vault read sys/policy/jokes-policy
Let’s now test our policy. We can have a token associated with our policy like this. We will try to create secret and access them. We will make sure that we don’t have any right on the super-joke secret.
$ vault token create -policy=jokes-policy
Key Value
--- -----
token s.RwWcjO5dGJRelpijZT6G7tTA
token_accessor 0f9YstNwuLT03iO7n8bZydwy
token_duration 768h
token_renewable true
token_policies ["default" "jokes-policy"]
identity_policies []
policies ["default" "jokes-policy"]
$ vault login
Token (will be hidden): (Enter the token returned above)
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.RwWcjO5dGJRelpijZT6G7tTA
token_accessor 0f9YstNwuLT03iO7n8bZydwy
token_duration 767h59m18s
token_renewable true
token_policies ["default" "jokes-policy"]
identity_policies []
policies ["default" "jokes-policy"]
$ vault kv get kv/secret/jokes/oneliner
====== Metadata ======
Key Value
--- -----
created_time 2020-03-06T16:26:37.861335217Z
deletion_time n/a
destroyed false
version 1
===== Data =====
Key Value
--- -----
goodone I dreamed I was forced to eat a giant marshmallow..
$ vault kv put kv/secret/jokes/test hello=world
Key Value
--- -----
created_time 2020-03-06T17:11:51.559299012Z
deletion_time n/a
destroyed false
version 1
$ vault kv put kv/secret/jokes/super-joke hello=world
Error writing data to kv/data/secret/jokes/super-joke: Error making API request.
URL: PUT https://127.0.0.1:8200/v1/kv/data/secret/jokes/super-joke
Code: 403. Errors:
* 1 error occurred:
* permission deniedWe can login with our username and password. We will be able to do all what the policy allows just like previously.
$ vault login -method=userpass \username=jeanblaguin \password=yolo$ vault kv put kv/secret/jokes/tada et=voila$ vault kv get kv/secret/jokes/oneliner$ vault kv get kv/secret/jokes/super-joke* permission denied
AppRole Auth
AppRole auth method is usefull when an app needs to obtain a Vault token with appropriate policies attached. Since each AppRole has attached policies, you can write fine-grained policies limiting which app can access which path.
We will create a policy and role for an app and get a role ID and Secret ID for our app to retrieve secrets from Vault. RoleID and SecretID are like a username and password that a machine or app uses to authenticate
As before we will enable this approle authentication, create a policy and a role linked to it. We do this through a shell inside the vault-0 POD. Don’t forget to login with your root token before running the commands. Note that you can do the same through the Web UI.
$ vault auth enable approlecreate a policy file named my-app-pol.hcl with following policies to set appropriate permissions.
# Read-only permission on 'secret/data/mysql/*' path
path "secret/data/mysql/*" {
capabilities = [ "read" ]
}create a my-app-pol policy and create a new AppRole my-app. The role token's time-to-live (TTL) is set to 12 hour and can be renewed for up to 24 hours of its first creation. We can attach several policies like this policies="my-app-pol,sql-pol,anotherpolicy"
$ vault policy write my-app-pol ./my-app-pol.hcl
$ vault write auth/approle/role/my-app token_policies="my-app-pol" \
token_ttl=1h token_max_ttl=4hSince the example created a jenkins role which operates in pull mode, Vault will generate the SecretID. Indeed Vault will generate the SecretID. You can set properties such as usage-limit, TTLs, and expirations on the SecretIDs to control its lifecycle. More information about approle
Let’s fetch the RoleID and SecretID of this role.The first command will retrieve the RoleID and the second command will generate a SecretID for the my-app role. We could have enforce a CIDR for the secret IDs to be used from specific set of IP addresses.
$ vault read auth/approle/role/my-app/role-id
Key Value
--- -----
role_id 6181e20e-74e1-104b-9566-f7c13d8332b0
$ vault write -f auth/approle/role/my-app/secret-id
Key Value
--- -----
secret_id f9e61734-a660-7d11-b65f-32b07c07ce24
secret_id_accessor 596b5262-ac8f-73ad-c5c1-7a1f486f2f3eTo login, use the auth/approle/login endpoint by passing the RoleID and SecretID.
$ vault write auth/approle/login role_id="6181e20e-74e1-104b-9566-f7c13d8332b0" \
secret_id="f9e61734-a660-7d11-b65f-32b07c07ce24"Key Value
--- -----
token s.UVZf8IdGo9ZxjbS0bPDrIYLD
token_accessor i3iRF1x8YZPTbb4KWAwuGaqo
token_duration 1h
token_renewable true
token_policies ["default" "my-app-pol"]
identity_policies []
policies ["default" "my-app-pol"]
token_meta_role_name my-app
Kubernetes Auth and Vault agent injector
Let’s take an app living in a kubernetes namespace that needs to retrieve some secrets like a username and password to access a database. In this example we will take advantage of the vault agent injector to retrieve the secret on behalf of the app and write that to a file that is mounted inside the POD. The injector will then stop running. In a next post we will see how we can instead ask vault to dynamically generate that database username and password and keep that secret renewed auto-magically.
To sum up the quite complex vault diagram , The app will authenticate to Vault through a kubernetes service-account. This service-account has previously been setup and attached to a policy in Vault. Once authenticated the app will ask for the username and password with the token issued by Vault. Actually our app will not talk to Vault directly, it is the vault agent injector sidecar that will take care of the authentication for the service and the secret retrieval.
All we have to do beside the service-account creation is to add some annotations to our POD for the vault-agent to work properly. The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request.
Vault configuration for kubernetes auth
Create a service account for vault
We already have some kubernetes service account defined by the chart installation. Vault kubernetes auth will use it to access our kubernetes cluster api to verify other service accounts like our app service account. Note that service accounts are namespaced so be sure to pay attention to the namespace where you create your service account.
$ kubectl get sa -n vault
NAME SECRETS AGE
default 1 5d2h
vault 1 4d21h
vault-agent-injector 1 4d21hCreate a service account for our app
Now we will create a specific service account named app-auth for our app that lives in the namespace app-ns
$ kubectl create namespace app-ns
$ kubectl create serviceaccount app-authCreate a policy for the app
Create a read-only policy, app-ro-pol in Vault. We will then add a secret to test the policy
# Create a policy file, app-ro-pol.hcl
$ tee app-ro-pol.hcl <<EOF
# As we are working with KV v2
path "kv/data/secret/app/*" {
capabilities = ["read", "list"]
}
EOF# Create a policy named app-ro-pol
$ vault policy write app-ro-pol app-ro-pol.hcl# add a secret
$ vault kv put kv/secret/app/config username='heisenberg' \
password='urdamnright' \
ttl='30s'
# get a token linked to the policiy
$ vault token create -policy=app-ro-pol
Key Value
--- -----
token s.QF6hHyuJ2IJ6SFafrjr5ibRn
token_accessor l4nbmoxhc149CeoDSMVoGQN4
token_duration 768h
token_renewable true
token_policies ["app-ro-pol" "default"]
identity_policies []
policies ["app-ro-pol" "default"]# login with the token
$ vault login
Token (will be hidden): #use token here below# test if we can access the secret
$ vault kv get kv/secret/app/config
====== Metadata ======
Key Value
--- -----
created_time 2020-03-09T12:25:43.144562975Z
deletion_time n/a
destroyed false
version 1====== Data ======
Key Value
--- -----
password urdamnright
ttl 30s
username heisenberg#test that we can't add a secret
$ vault kv put kv/secret/app/config username='gandalf' \
password='ushallnotpass' \
ttl='30s'
...
* 1 error occurred:
* permission denied
Configure Kubernetes auth method
Now, enable and configure the Kubernetes auth method. You need to be login as root to do that.
# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
$ vault auth enable kubernetes# Finding your api-server url. Depending on your kube config adapt the cluster index
$ KUBERNETES_TCP_ADDR=$(kubectl config view -o jsonpath="{.clusters[0].cluster.server}")# Tell Vault how to communicate with the Kubernetes cluster
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=${KUBERNETES_TCP_ADDR} \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crtSuccess! Data written to: auth/kubernetes/config# Create a role named, 'app' to map our app's Kubernetes Service Account to
# Vault policies and default token TTL
$ vault write auth/kubernetes/role/app \
bound_service_account_names=app-auth \
bound_service_account_namespaces=app-ns \
policies=app-ro-pol \
ttl=24h
Verify the Kubernetes auth method configuration (optional)
In a terminal with kubectl configured create a Pod with a container running alpine:3.7 image.
$ kubectl config set-context --current --namespace=app-ns
$ kubectl run --generator=run-pod/v1 tmp --rm -i --tty --serviceaccount=app-auth --image alpine:3.7Once you are inside the container, install cURL and jq tools.
/$ apk update && apk add curl jqSet the VAULT_ADDR environment variable to point to the running Vault where you configured Kubernetes auth method, and test the connection
/$ VAULT_ADDR=https://vault-server.vault.svc.cluster.local:8200
/$ curl -ks $VAULT_ADDR/v1/sys/health | jq
{
"initialized": true,
"sealed": false,
"standby": false,
"performance_standby": false,
"replication_performance_mode": "disabled",
"replication_dr_mode": "disabled",
"server_time_utc": 1543969628,
"version": "1.0.0+ent",
"cluster_name": "vault-cluster-e314942e",
"cluster_id": "2b4f6213-d58f-0530-cf07-65ea467181f2"
}Set KUBE_TOKEN to the service account token value:
/$ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/$ echo $KUBE_TOKENNow, test the kubernetes auth method to ensure that you can authenticate with Vault.
/$ curl -k --request POST \
--data '{"jwt": "'"$KUBE_TOKEN"'", "role": "app"}' \
$VAULT_ADDR/v1/auth/kubernetes/login | jq
{
...
"auth": {
"client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
"accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
"policies": [
"default",
"app-ro-pol"
],
"token_policies": [
"default",
"app-ro-pol"
],
"metadata": {
"role": "app",
"service_account_name": "app-auth",
"service_account_namespace": "app-ns",
"service_account_secret_name": "app-auth-token-5cj2w",
"service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
},
"lease_duration": 86400,
"renewable": true,
"entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
"token_type": "service"
}
}Success ! client_token is generated and app-ro-pol policy is attached with the token. The metadata displays that its service account name (service_account_name) is app-auth.
Injecting Vault Secret into the POD
With the Kubernetes auth method configured on the Vault server, it is time to spin up a deployment which leverages Vault Agent to automatically authenticate with Vault and retrieve the secret.
Let’s first create a deployment inside our app-ns namespace
$ cat <<EOF >> app.yaml
# app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
namespace: app-ns
labels:
app: vault-agent-demo
spec:
selector:
matchLabels:
app: vault-agent-demo
replicas: 1
template:
metadata:
annotations:
labels:
app: vault-agent-demo
spec:
serviceAccountName: app-auth
containers:
- name: app
image: jweissig/app:0.0.1
EOF
$ kubectl create -f app.yamlNext, let’s launch our example application and create the service account. We can also verify there are no secrets mounted at /vault/secrets.
$ kubectl create -f app.yaml
$ kubectl exec -ti app-XXXXXXXXX -c app -- ls -l /vault/secretsHere is an annotation patch we can apply to our running example application’s pod configuration that sets specific annotations for injecting our secret/app/config Vault secret.
cat <<EOF >> patch-app.yaml
# patch-basic-annotations.yaml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
vault.hashicorp.com/agent-inject-secret-app-config: "kv/data/secret/app/config"
vault.hashicorp.com/role: "app"
EOFThese annotations define a partial structure of the deployment schem and are prefixed with vault.hashicorp.com.
agent-injectenables the Vault Agent injector serviceroleis the Vault Kubernetes authentication roleroleis the Vault role created that maps back to the K8s serviceaccount
Notice the vault.hashicorp.com/ca-cert annotation, the Vault client (injected inside an init container named vault-agent-init) is connecting to Vault which is protected with TLS. The client doesn't just trust the server's certificate authenticity, so it needs to have the CA certificate that the server certificates were signed with to double check.
Simply add the CA certificate used to sign Vault servers certificates. You can see an example of the annotations here (note: this uses client certs as well for TLS, where the server also verifies the authenticity of the clients certs, but we aren’t enforcing client certs here).
Next, let’s apply our annotations patch.
kubectl patch deployment app --patch "$(cat patch-app.yaml)"Verifying the secret
If it all goes well you should now see a tmpfs mount at /vault/secrets and a config secret containered in there. What happened here, is that when we applied the patch, our vault-k8s webhook intercepted and changed the pod definition, to include an Init container to pre-populate our secret, and a Vault Agent Sidecar to keep that secret data in sync throughout our applications lifecycle.
$ kubectl exec 7f9d49d495-z24l7 --container app -- cat /vault/secrets/app-config
data: map[password:urdamnright ttl:30s username:heisenberg]
metadata: map[created_time:2020-03-09T12:25:43.144562975Z deletion_time: destroyed:false version:1]Updating the secret
As there is a side car if we modify the secret, the update will be automatically reflected to our POD. Let’s try by updating our secret
$ vault kv put kv/secret/app/config username=walter password="I’m in the empire business."Wait till the TTL expires and check the secret again to see the changes
$ kubectl exec 7f9d49d495-z24l7 --container app -- cat /vault/secrets/app-config
data: map[password:I’m in the empire business. ttl:30s username:walter]
metadata: map[created_time:2020-03-09T12:25:43.144562975Z deletion_time: destroyed:false version:1]Without the sidecar always running
Even though you can limit the memory and CPU limit of the side car through annotations (agent-limits-memory, agent-limits-cpu), there are smoe use cases when you don’t want a side car always running.
If you just want to retrieve the secret once and mount it to your POD you can use the agent-pre-populate-only annotation and the init container will be the only injected container. If true, no sidecar container will be injected at runtime of the pod. Let's try
cat <<EOF >> patch-pre-polutate-only-app.yaml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
vault.hashicorp.com/agent-inject-secret-app-config: "kv/data/secret/app/config"
vault.hashicorp.com/role: "app"
vault.hashicorp.com/agent-pre-populate-only: "true"
EOF
$ kubectl patch deployment app --patch "$(cat patch-pre-polutate-only-app.yaml)"
# the init container starts
$ kubectl get po
NAME READY STATUS RESTARTS AGE
app-9bd67fcfc-8488k 0/1 Init:0/1 0 1s
# only our POD is running
$ kubectl get po
NAME READY STATUS RESTARTS AGE
app-9bd67fcfc-8488k 1/1 Running 0 7s
$ kubectl exec -ti app-9bd67fcfc-8488k -c app -- cat /vault/secrets/app-config
data: map[password:I’m in the empire business. username:walter]
metadata: map[created_time:2020-03-13T19:59:34.956698063Z deletion_time: destroyed:false version:3]Troubleshooting
if it doesn’t work make sure the injector side car launched you should see a ready 2/2 on the app POD. You can also check the vault-agent-init container logs kubectl logs app-XXXXXX vault-agent-init. As well as the the Vault logs itself kubectl logs -n vault vault-0
You can set the ENV var AGENT_INJECT_LOG_LEVEL to debug in the vault-agent-injector deployment to get more details in the log. Then you can view the agent-injector logs kubectl logs -n vault vault-agent-injector-XXXXXX
If you encounter the error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate is valid for vault-server, vault-server.vault, vault-server.vault.svc, vault-server.vault.svc.cluster.local, not vault.vault.svc" backoff
it is because the injector try to find the vault server at vault.vault.svc instead of vault-server.vault.svc.cluster.local. Do not modify the chart injector.externalVaultAddr value as it will remove your vault-o pod.
Indeed as stated in the chart , setting this will disable deployment of a vault server along with the injector. In the chart template we see that by default the value is <vault.scheme>://<vault.fullname>.<Release.Namespace>.svc:<server.service.port>. Just modify the ENV variable in the deployment
You can also launch a debug POD in your app namespace to make sure you can login with kubernetes authentication to vault through the HTTPS API.
$ kubectl run --generator=run-pod/v1 -i --tty tmp2 --image=praqma/network-multitool --serviceaccount=app-auth --restart=Never -- sh
$ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/t
oken)
$ VAULT_ADDR=https://vault-server.vault.svc:8200
$ ln -s /run/secrets/kubernetes.io/serviceaccount/ca.crt ca.crt
$ curl --cacert ca.crt --request POST \
> --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "app"}' \
> $VAULT_ADDR/v1/auth/kubernetes/login | jq
{
"request_id": "da430f47-1535-3eb3-2199-13b3c0f9dd52",
"lease_id": "",
"renewable": false100 1584 ,
"lease_duration": 0,
"data": null,
"wrap_info": null,
"warnings": null,
"auth": {
"client_token": "s.1zKhdhkvxnI54DMJxVt",
"accessor": "R5biZXHC254D"
"app-ro-pol",
"default"
],
"token_policies": [
100 657 100 927 12634 17826 "app-ro-pol",
"default"
],
"metadata": {
"role": "app",
"service_account_name": "app-auth",
"service_account_namespace": "app-ns",
"service_account_secret_name": "app-auth-token-5cj2w",
"service_account_uid": "4f3832f5-2fba-4067-bf75-f9e530940ca8"
},
"lease_duration": 3600,
"renewable": true,
"entity_id": "c2ddd210-9b17-3aec-3782-0318d307db13",
"token_type": "service",
"orphan": true
}
}Apply a template to the injected secrets
The secret is present on the container. However, the structure is not in one expected by the application.
let’s add some connection information to our secret. Note that you can also do that through the web UI.
$ vault kv put kv/secret/app/config username=walter password=sayMyName database=baking_bad port=27018You can format your secret data using by leveraging Vault Agent Templates, which is very useful for dealing with your various output formatting needs. In the next example here, we are going to parse our secrets data into a Mongo DB connection string.
cat <<EOF >> patch-mongo-template-app.yaml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-app-config: kv/data/secret/app/config
vault.hashicorp.com/agent-inject-status: update
vault.hashicorp.com/agent-inject-template-app-config: |
{{- with secret "kv/data/secret/app/config" -}}
mongo://{{ .Data.data.username }}:{{ .Data.data.password }}@mongo:{{ .Data.data.port}}/{{ .Data.data.database}}
{{- end -}}
vault.hashicorp.com/ca-cert: /run/secrets/kubernetes.io/serviceaccount/ca.crt
vault.hashicorp.com/role: app
EOF
$ kubectl patch deployment app --patch "$(cat patch-mongo-template-app.yaml)"
$ kubectl exec -ti app-9bd67fcfc-8488k -c app -- cat /vault/secrets/app-config
mongo://walter:sayMyName@mongo:27018/baking_badGoing Further
Now that we have a good understanding of Vault and secret injection in kubernetes, it might be interesting to take a look at dynamic secrets for database.
The idea is to give a root password to a database to vault and let it rotate the credential. You can also create database roles to control the tables to which a user has access and the lifecycle of the credentials.
At the end Vault takes care of the database credentials and secrets rotation. If the credentials leak, the blast radius is dramatically reduced as the period of usefulness for credentials is limited. When a human operator is managing credentials they must manually be revoked, that is assuming the operator is aware of the leak, often they are not until it is too late.
If you have questions remarks you can PM me: telegram:@Zgorizzo mail: ben@cogarius.com
