top doesn’t agree with influx. top is showing kubelet between 10-30%. influx is showing CPU bursts of up to 100%. Both are probably correct though, since influx is tracking total CPU usage while top we’re just using kubelet for instrumentation.

Kabana reveals a new compalint in the logs! Perhaps we’ll have better luck on this adventure before I return to work on the others. This might also be the underlying cause of the kube-state-metrics repeatedly dieing off also.

E0425 23:30:20.198117       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.0.0-20191109102209-3c0d1af94be5/tools/cache/reflector.go:108: Failed to list *v1beta2.VerticalPodAutoscaler: the server could not find the requested resource (get verticalpodautoscalers.autoscaling.k8s.io)

VerticalPodAutoscaler will modify the requests fields within a lower bounds to the the specified limit as based on the pod’s need. There is not really a defacto recommended Helm chart for install the VirticalPodAutoscaler, just running scripts from the repository. I am torn between trying to disable the VPA in the Prometheus configuration or pushing forward.

Giving it a bit of the old college try before re-evaluating. Cloning the repository resulted in nearly 100MB download. On closer inspection the yaml files within the vertical-pod-autoscaler/deploy directory appear to have the correct version of the resources within them. I feel like this is preferable.

COMPONENTS="vpa-v1-crd vpa-rbac updater-deployment recommender-deployment admission-controller-deployment"
for i in $COMPONENTS; do
    kubectl apply -f "https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/deploy/$i.yaml"
done

Simple script to get these installed and going. Checking the logs…and…nnooooo!

E0426 03:32:03.137370       1 updater.go:114] Error getting Admission Controller status: leases.coordination.k8s.io "vpa-admission-controller" not found. Skipping eviction loop

Maybe I can just disable it for now.