k8s: Fixing My Broken Cluster II
• Mark Eschbach
top doesn’t agree with
top is showing
kubelet between 10-30%.
influx is showing CPU bursts of up to
100%. Both are probably correct though, since
influx is tracking total CPU usage while
top we’re just using
kubelet for instrumentation.
Kabana reveals a new compalint in the logs! Perhaps we’ll have better luck on this adventure before I return to work on
the others. This might also be the underlying cause of the
kube-state-metrics repeatedly dieing off also.
E0425 23:30:20.198117 1 reflector.go:156] email@example.com/tools/cache/reflector.go:108: Failed to list *v1beta2.VerticalPodAutoscaler: the server could not find the requested resource (get verticalpodautoscalers.autoscaling.k8s.io)
VerticalPodAutoscaler will modify
requests fields within a lower bounds to the the specified limit as based on the pod’s need. There is not really
a defacto recommended Helm chart for install the VirticalPodAutoscaler, just running scripts from the repository. I am
torn between trying to disable the VPA in the Prometheus configuration or pushing forward.
Giving it a bit of the old college try before re-evaluating. Cloning the repository resulted in nearly 100MB download.
On closer inspection the yaml files within the
vertical-pod-autoscaler/deploy directory appear to have the correct
version of the resources within them. I feel like this is preferable.
COMPONENTS="vpa-v1-crd vpa-rbac updater-deployment recommender-deployment admission-controller-deployment" for i in $COMPONENTS; do kubectl apply -f "https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/deploy/$i.yaml" done
Simple script to get these installed and going. Checking the logs…and…nnooooo!
E0426 03:32:03.137370 1 updater.go:114] Error getting Admission Controller status: leases.coordination.k8s.io "vpa-admission-controller" not found. Skipping eviction loop
Maybe I can just disable it for now.