Exploring Banzi Cloud's Vault HA using GCP
• Mark Eschbach
The goal for today: get Vault configured in HA mode utilizing Google’s Cloud Storage System while running within Kubernetes. Once you create a new service account within GCP you can give it the roles for object creation and object viewing. I will worry about scoping and tightening these rules later. I hope out of the box these rules provided scope the account to the project it was created within.
In exchange for creating the service account one is given a
json file. This is imported into Kubernetes via
kubectl create secret generic vault-gcp-service-account --from-file=service-account.json.json=gke-vault-test-14338aab3c4d.json.
I wonder what other secret types exist beyond
serviceaccount. Added the following fragment to the
spec: credentialsConfig: env: GOOGLE_APPLICATION_CREDENTIALS path: /etc/gcp/service-account.json secretName: vault-gcp-service-account
When Kubernetes does not start the pod, a
kubectl describe pod vault-0 gives a very detailed description as to why.
However to get more detail sometimes a
kubectl logs vault-0 vault might also be needed. The container logs show the
secret isn’t being properly mounted within the container:
time="2019-02-11T17:00:56Z" level=fatal msg="error creating kv store: error creating google cloud storage kv store: error creating gcs client: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: read /etc/gcp/service-account.json: is a directory"
vault-operator was out of date?
helm list produced this:
illmannered-hound 1 Mon Feb 4 14:50:08 2019 DEPLOYED vault-operator-0.2.4 0.4.3 default
helm delete illmannered-hound and
helm install banzaicloud-stable/vault-operator the following list was given:
knobby-pika 1 Mon Feb 11 10:13:48 2019 DEPLOYED vault-operator-0.2.6 0.4.8 default
The change logs are effectively the commit sets in the releases,
with no change logs or anything else. This makes it difficult to understand what has changed from a user perspective.
However from my guess is we are now running
bank-vaults:0.4.8. Until I spin up a new cluster and find the 3
n1-standard-1 (1 vCPU, 3.75 GB memory)
are insufficient to deploy the resources now. The nodes appear to have 20% of their resources consumed by cluster
orchestration. Increasing the pool with
gcloud container clusters resize vault-test-cluster-1 --node-pool default-pool --size 6 --zone us-west1
brought my total up to 18….and the same log message is still presented.
With the resources are now being properly created via Vault however the
bank-vaults process is continuing to trip on
the unseal keys. Switching gears for a while.