Picking up from yesterday the Vault instance wasn’t unsealed. The pod vault-0 container banks-vaults shows:

time="2019-02-05T05:11:06Z" level=info msg="checking if vault is sealed..."
time="2019-02-05T05:11:06Z" level=info msg="vault sealed: true"
time="2019-02-05T05:11:06Z" level=error msg="error unsealing vault: unable to get key 'vault-unseal-0': error getting secret for key 'vault-unseal-0': secrets \"vault-unseal-keys\" is forbidden: User \"system:serviceaccount:default:default\" cannot get secrets in the namespace \"default\""

Hmm, I think I misunderstood something in the resource specification. Looks like I need to create the role specified in the resource definition. Makes total sense but I was going for simplicity of implementation. Now to make it happen!

Based on the error message I believe the container is running under the credential system:serviceaccount:default:default. kubectl describe pod vault-0 did not contain any references to the role. Neither did kubectl get rolebinding --namespace=kube-system :-(.

Demystifying RBAC in Kubernetes defines two types of roles: Users and Service Accounts. Service Accounts are intra-cluster resources while users are located outside of the pod. There was a lot of great details on the principal (subject), action (verbs), and target (api resources). A ClusterRole is cluster wide while the Role is scoped to a namespace. A RoleBinding or ClusterRoleBinding is utilized to give a subject, either a group or specific role, a binding.

Okay! After a whole bunch of reading I think I finally figured it out. system:serviceaccount is stating the named resource is a service account. The next section default is referring to the default namespace. While the last element in the : path is the named service account under the default namespace. According to this hypothesis the results I should be able to to list the account with kubectl get serviceaccounts --namespace=default. kubectl describe serviceaccounts default provides the following:

Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   default-token-svm5r
Tokens:              default-token-svm5r
Events:              <none>

There are many interesting fields here however the Mountable secrets & Tokens look most interesting. Looks like the OpenShift project confirms my hypothesis of the URI, so that is always exciting.

Turns out you can edit service accounts using something like kubectl describe serviceaccounts default. This will allow one to add the vault-unseal-keys to the list of mountable secrets. This however was insufficient as I think this only means it will expose an existing one and can be confirmed as missing using kubectl describe serviceaccounts default shows it is missing.

Tune in tomorrow for either get Vault established or shaving more yaks :-) !