Originally I tried to run promtheus-operator on bare metal. After multiple hours attempting to debug the underlying issues I gave up. The weather was getting warmer so spending CPU cycles on something which I could not get to work properly was something I wanted to avoid. I figured I would return at a later time. After I learned more about running Kubernetes on bare metal.

Well, the time has come. This time I am going to avoid prometheus-operator in favor of going a layer lower. I would like to run an instance of Prometheus on local volumes. Let’s see how it shakes out

I’ll be using helm’s prometheus chart. First up is creating local persistent volumes. Something along the lines of:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    "app.kubernetes.io/part-of": "platform"
    "app.kubernetes.io/managed-by": "meschbach"
    "app.kubernetes.io/component": "monitoring"
  name: machine-platform-promteheus-alertmanager-0
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  local:
    path: /storage/machine-platform-promteheus-alertmanager-0
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - machine
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    "app.kubernetes.io/part-of": "platform"
    "app.kubernetes.io/managed-by": "meschbach"
    "app.kubernetes.io/component": "monitoring"
  name: machine-platform-promteheus-pushgateway-0
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  local:
    path: /deep/ark/gluster-brick/machine-platform-promteheus-pushgateway-0
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - machine
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  volumeMode: Filesystem

machine is replaced with the name of the node the PVs should exist on. The following brought up a prometheus instance which was working to collect node details and API server information.

alertmanager:
  persistentVolume:
    enabled: false
    size: 1Gi
    storageClass: 'local-storage'
  statefulSet:
    enabled: true
pushgateway:
  persistentVolume:
    enabled: false
    storageClass: 'local-storage'
    size: 1Gi
server:
  persistentVolume:
    enabled: false
    size: 1Gi
    storageClass: 'local-storage'
  statefulSet:
    enabled: true

Now to figure out how to get the PVs to mount properly.