Deploying Prometheus on Kubernetes
• Mark Eschbach
Time to get some metrics out the clusters. I needs me some details.
Detour o using Terraform
Originally I desired to use Terraform for managing deployments. This worked well for the layer below, finalizing the configurations of the underlying platform. With my irrational fear of YAML I really desired to stay away from it. Alas, the community has spoken and we’ll just become YAML engineers during this generation of technologies.
Reviewing the current state of Helm Security
As last I left it, Helm seemed to have been moving away from having agents deployed into the cluster. According to the current documentation Helm still uses Tiller instance. The stance of secure enough is little bothersome since I had such great problems securing them at the last place. Although this might be a result of my inexperience with the platform.
The basic answer, prior to Helm 3 (not released yet), is to create a new service account for each domain you would like deployed with Tiller. The service account would correctly allow isolation of the Tiller instances. This solves limiting the capabilities of a particular Tiller instance however this does not resolve restricting who may interact with the Tiller instance.
To properly authorization requests in a multi-role environment the recommendation is to create a certificate authority for each domain with certificate verification. This isn’t the end of the world since we could easily create a new CA via Vault. However this results in further complexity for bootstrapping. As per my normal policy a developer should be able to create a fully formed local environment with relative ease. Additionally, I would rather not have a process consuming resources which is rarely used with a large security surface.
So far running Tiller locally appears to be the best option available. There are several questions which need to be answered though:
- How are credentials distributed to this agent?
- How is state management handled?
- Will this integrate into CI well?
Running Tiller locally states the process needs the following options: -storage=secret --listen localhost:44134
. A set of Google searches did not yield any
results documenting what the Tiller flags are or how they are used. Digging in I found the Tiller flags in source
which will work against the Kubernetes secret storage
in the namespace as defined in TILLER_NAMESPACE or DefaultTillerNamespace (kube-system
).
This seems like a more sane method of controlling access since the access control mechanisms will exist for the users
anyway. Meaning beyond the delivery pipeline the users will not have access to more than their current user allows.
Navigating with a local Tiller
A lot of talk, no action. Let’s do something.
TILLER_NAMESPACE=platform-monitoring tiller --storage=secret --listen localhost:44143 &
TILLER_PID=$!
function cleanup() {
echo "Terminating Tiller $@"
kill $TILLER_PID
}
trap cleanup SIGTERM SIGINT SIGHUP EXIT
export HELM_HOST=localhost:44143
This will launch Tiller under the current kubectl config get-context
. I would hope the context is captured instead
of using a file however I am not that sure. Next up is to ask it to install the components:
echo "Ensuring Helm is configured"
if [ -d $HOME/.helm ]; then
echo "Helm already existing"
else
helm init --client-only
fi
echo "Updating the repository"
helm repo update
echo "Install prometheus"
helm install stable/prometheus-operator --name prometheus-operator --namespace platform-monitoring --tiller-connection-timeout 5
After a few minutes this will complete launching the system within the name space platform-monitoring
. When the
software is ready you can run kubectl proxy --port=8090
to access Prometheus at http://localhost:8090/api/v1/namespaces/platform-monitoring/services/prometheus-operator-prometheus:9090/proxy.
Granted there are some problems given it’s expected to be served from the root however that isn’t too bad. First thing
on my queue for tomorrow.