Use local-path-provisioner for PostGreSQL

Table of Contents

Overview

We offer two different storage types at SysElevenStack: Block Storage via Cinder and Local SSD Storage on your worker nodes. The Block Storage is the default storage class.
There are restrictions for local SSD storage as we describe on a background article here. The benefits of IO perfomance is usually the reason to still use it.
For this use case, MKA provides a Building Block called local-path-provisioner. This Building Block creates a new storage class which can be set up on local SSD Storage.
Bear in mind that it does not solve the situation of losing data, when a node migration starts, but it gives you control on how to handle it.

Prerequisites

You need to be familiar with the following tasks:

  1. create a group of worker nodes with local storage
  2. add labels and taints to your worker nodes
  3. deploying the local-path-provisioner
  4. deploying the pg-operator

Adding labels and taints to your worker nodes

For this example you need to set labels and taints. To make sure that the database is deployed to the worker nodes with local storage it needs to be explicitly described via taints, tolerations and affinities.
The label set on your machinedeployment is pgo-cluster: "true" while the taint is node.syseleven.de/storage=local:NoSchedule

**Please make sure you have at least two different machine deployments with local storage. This serves the purpose to have time to react, when you upgrade all the worker nodes and you can plan your update strategy accordingly. The reason for this is to adjust the volumes as described. One machine deployment per desired PostgreSQL-Instance is recommended.

Configure tolerations and affinities to Postgres

Now the Building Block needs to be configured to use only those nodes, which are correctly labelled. For this we use the following configuration:

#values-pg-db.yaml
backup:
  volumeSpec:
    storageclass: sys11-quobyte-external-provisioner
    size: 10G
pgPrimary:
  tolerations: '[{ "key": "node.syseleven.de/storage", "effect": "NoSchedule", "value": "local", "operator": "Equal" }]'
  volumeSpec:
    size: 10G
    storageclass: "local-path"
  affinity:
    nodeAffinityType: "required"
    nodeLabel:
      pgo-cluster: "true"
replicas:
  size: 2
  volumeSpec:
    size: 10G
    storageclass: "local-path"
disable_telemetry: true

The toleration allows the pods to be deployed to the local storage worker nodes, while the affinity enforces the usage of the local storage worker nodes. Also, the storageclass "local-path" is the preconfigured storageClassname of the Building Block local-path-provisioner.

Updating your MetaKube Cluster in waves

As local storage does not save any data, it will be lost during a Kubernetes- or OS-update of those worker nodes. To prevent data loss and to have time to react, it is recommended to update machine deployments sequentially. When you update a worker node, the pods of this worker node will be evicted and rescheduled to the new one. After the reschedule, you need to delete the Persistent Volume Claim and recreate it.

# get the names of the pending pods and its volumeClaim
kubectl get pods --field-selector status.phase=Pending -o yaml | grep persistentVolumeClaim -A1
      persistentVolumeClaim:
        claimName: pg-db-repl1
#pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    pg-cluster: pg-db
    vendor: crunchydata
  name: pg-db-repl1 #rename pvc accordingly
  namespace: syseleven-pg-operator
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10G #adjust volume size accordingly
  storageClassName: local-path
# delete pvc and recreate it
kubectl delete -f pvc.yaml && kubectl apply -f pvc.yaml

After the pod is up and running you can start updating the next machine deployment.