The goal for today: get Vault configured in HA mode utilizing Google’s Cloud Storage System while running within Kubernetes. Once you create a new service account within GCP you can give it the roles for object creation and object viewing. I will worry about scoping and tightening these rules later. I hope out of the box these rules provided scope the account to the project it was created within.

In exchange for creating the service account one is given a json file. This is imported into Kubernetes via kubectl create secret generic vault-gcp-service-account --from-file=service-account.json.json=gke-vault-test-14338aab3c4d.json. I wonder what other secret types exist beyond generic and serviceaccount. Added the following fragment to the Vault CRD.

spec:
  credentialsConfig:
    env: GOOGLE_APPLICATION_CREDENTIALS
    path: /etc/gcp/service-account.json
    secretName: vault-gcp-service-account

When Kubernetes does not start the pod, a kubectl describe pod vault-0 gives a very detailed description as to why. However to get more detail sometimes a kubectl logs vault-0 vault might also be needed. The container logs show the secret isn’t being properly mounted within the container:

time="2019-02-11T17:00:56Z" level=fatal msg="error creating kv store: error creating google cloud storage kv store: error creating gcs client: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: read /etc/gcp/service-account.json: is a directory"

Perhaps the vault-operator was out of date? helm list produced this:

illmannered-hound	1       	Mon Feb  4 14:50:08 2019	DEPLOYED	vault-operator-0.2.4	0.4.3      	default

When helm delete illmannered-hound and helm install banzaicloud-stable/vault-operator the following list was given:

knobby-pika	1       	Mon Feb 11 10:13:48 2019	DEPLOYED	vault-operator-0.2.6	0.4.8      	default

The change logs are effectively the commit sets in the releases, with no change logs or anything else. This makes it difficult to understand what has changed from a user perspective. However from my guess is we are now running bank-vaults:0.4.8. Until I spin up a new cluster and find the 3 n1-standard-1 (1 vCPU, 3.75 GB memory) are insufficient to deploy the resources now. The nodes appear to have 20% of their resources consumed by cluster orchestration. Increasing the pool with gcloud container clusters resize vault-test-cluster-1 --node-pool default-pool --size 6 --zone us-west1 brought my total up to 18….and the same log message is still presented.

With the resources are now being properly created via Vault however the bank-vaults process is continuing to trip on the unseal keys. Switching gears for a while.