Deploy a development setup

This tutorial will guide you through a fully functional Kubernetes cluster without your team doing any global configuration steps during their own application development.
This includes plugins and additional components like:

  1. Ingress controller (ingress-nginx)
  2. TLS certificate management (cert-manager)
  3. Monitoring (Grafana, Prometheus and Loki)
  4. General purpose in-memory database (redis-ha)

At the end of this tutorial you

  • have a running Kubernetes cluster
  • installed all necessary Building Blocks for your developers to start immediately
  • have an examplatory ingress defined for Grafana
  • only need to focus on application definition and development while all underlining tasks are either configured or easy to use

Predefinitions and Preparations

What you need

To finish this tutorial successfully you need to have a few local tools preinstalled, we mentioned on Getting Started.

Besides the tools you also need a DNS provider with API-Endpoints, e.g. SysEleven DNS or other providers.
Specifically, you need a DNS provider which is supported by external-dns to manage your subdomain records.

Configure GitLab Settings

Make sure you are familiar with the basic concept by doing our Getting Started tutorial.

First create a MetaKube Cluster. You need at least three nodes of the flavor m1c.medium. Download the kubeconfig and copy it to the path according to your kubectl installation (e.g. ~/.kube/config).
After this login to code.syseleven.de. To centralize all reoccuring tasks we recommend using one repository for CI/CD purposes.
Create a git repository defined as $customer/$project, e.g. "syseleven/my-project". We will refer to this repository as control repository, as we control all necessary tasks regarding the setup on MetaKube from here out.
Adjust the settings as described in setup a control repository and add your kubeconfig as CI/CD variable accordingly. Our variable KUBECONFIG will be assigned to the environment "development".

Configure CRDs on MetaKube Cluster

Several of our Building Blocks define custom resources (CRDs). Some also depend on resource definitions from other Building Blocks, leading to a circular dependency that needs to be resolved. The easiest way to resolve this dependency is to deploy the core CRDs before deploying any of the Building Blocks.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm pull --untar --untardir /tmp/kube-prometheus-stack --version 13.12.0 prometheus-community/kube-prometheus-stack
kubectl apply -f /tmp/kube-prometheus-stack/kube-prometheus-stack/crds/
# rm -r /tmp/kube-prometheus-stack

Start Structuring your Control Repository

Clone your control repository locally. We assume you clone the repository onto the local path ~/git/my-project.:

cd ~/git
export customer=set-your-namespace
git clone git@code.syseleven.de:$customer/my-project.git

We will work on the main branch for now. After this tutorial you can change the environment settings to adjust your branches as you need to.
The directory structure will be as follows:

/
└── syseleven-kube-prometheus-stack/
    ├── .gitlab-ci.yml
    ├── values-kube-prometheus-stack-development.yaml
└── syseleven-loki-promtail/
    ├── .gitlab-ci.yml
└── syseleven-ingress/
    ├── .gitlab-ci.yml
└── syseleven-external-dns/
    ├── .gitlab-ci.yml
    └── values-external-dns.yaml
└── syseleven-cert-manager/
    ├── .gitlab-ci.yml
└── syseleven-redis-ha/
    ├── .gitlab-ci.yml
    └── values-redis-ha.yaml
└── syseleven-pxc-operator/
    ├── .gitlab-ci.yml
    ├── values-pxc-db-development.yaml
    └── values-pxc-db.yaml
.gitlab-ci.yml

On the root path lies a .gitlab-ci.yml and a subfolder for each Building Block. This ensures a better reading structure. Each subfolder expects its own .gitlab-ci.yml where the repository of the corresponding Building Block will be included. An example file follows underneath:

include:
  - project: syseleven/building-blocks/helmfiles/cert-manager
    file: JobDevelopment.yaml
    ref: 4.8.2

The project refers to the repository residing on [code.syseleven.de]. The file defines the environment, which can be development, staging or production. The ref defines the appropriate branch at which the specific version can be referenced to. The main branch is hereby the latest branch.

The global pipeline definition will be written in the main directory also as .gitlab-ci.yml and needs two stages defined: diff and deploy. Those are necessary for the Building Blocks, as they define jobs in those two stages. See further beneath in the script on how to write it in your .gitlab-ci.yml.
For each Building Block you can look at our Building Block Catalogue to see, what to add to the Building-Block-specific .gitlab-ci.yaml and values.yaml for each one in the subfolders.

You can use this script to create the basic structure for this tutorial. Depending on the variables bbList , project and environments it generates necessary folders and files and adds the configurations for a basic development environment. It will not include them in the global /.gitlab-ci.yml so you can choose later, which Building Blocks you want to deploy:

#!/bin/sh

#save this e.g. as ~/git/repo-structure.sh and use the command
# bash ~/git/repo-structure.sh

bbList="kube-prometheus-stack loki-promtail cert-manager ingress-nginx external-dns pxc-operator redis-ha"
project="$HOME/git/my-project"
environments="development"

echo -n "stages:
  - diff
  - deploy" > $project/.gitlab-ci.yml
for i in $bbList; do
  echo "provisioning directory path syseleven-$i"
  mkdir $project/syseleven-$i
  echo -n "include:
    - project: syseleven/building-blocks/helmfiles/$i
      file: JobDevelopment.yaml
      ref: main" > $project/syseleven-$i/.gitlab-ci.yml
#add empty values*.yaml files if you want to customize right away
#    touch $project/syseleven-$i/values-$i.yaml
#  if [ "${#environments[@]}" -eq 0 ]; then
#    echo "enviroments not set"
#  else
#    for j in $environments; do
#      touch $project/syseleven-$i/values-$i-$j.yaml
#    done
#  fi
done

After creating the directory structure, check for our naming convention as this is needed for the Building Blocks to select the proper files for Deployment.

The SysEleven Building Block naming convention for directories is:
/
└──syseleven-$BUILDING-BLOCK/
and for file names
├──values-$BUILDING-BLOCK[-$ENVIRONMENT].yaml

Enable Building Blocks Step by Step

Deploy the Stack for Reverse Proxying

After setting up the folder structure we will now enable the Building Blocks incrementally.

At first we start with the Kubernetes reverse proxy stack. This includes:

  1. ingress-nginx
  2. cert-manager
  3. external-dns

This manages our DNS entries for our domain (external-dns), the certificates via Let's Encrypt (cert-manager) and gives our cluster the ability to create and manage ingress definitions (ingress-nginx).

While the Building Blocks ingress-nginx and cert-manager do not need any specific configuration to work (those will be set automatically due to the environment it works in, e.g. a floating IP for your LoadBalancer service) the external-dns needs a file where you define your credentials like an API token for managing the DNS records of your provider. If you use SysEleven DNS Zones, then you can add the file syseleven-external-dns/values-external-dns.yaml with the following content:

designate:
  username: "$sysElevenStackUser"
  password: "$sysElevenStackPassword"
  projectName: "the SysEleven Stack project name"

After you saved this file, add the Building Blocks to your global pipeline configuration, which will be written in .gitlab-ci.yml. It should now look like this:

stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml

Now it is set and done. Push the changes to your control repository

#make sure you're inside ~/git/my-project
git add .
git commit -m "add reverse proxy BBs"
git push

and watch the pipeline work on https://code.syseleven.de/$customer/my-project/-/pipelines!

The Ingress Controller needs some time to allocate a floating IP for its service. Check it with kubectl get svc -n syseleven-ingress-nginx -wand wait for the pending state to change to a valid IP!

Deploy Obersavibility Stack with an Ingress Resource for Grafana

After the pipeline finished we will start to add the observability stack. This includes tools like Prometheus for metrics, Loki/Promtail for logs, the Alertmanager for alerts and Grafana as GUI. We will also set an ingress for Grafana to be able to start right off seeing all necessary metrics.

For this to work we need to adjust the tooling a bit more. The kube-prometheus stack is highly customizable and would need configuration changes in a production environment, for example alerting rules. Both Loki and promtail rarely require customization. You can find all customization options in the description of the Building Block. For this tutorial, we only need to configure an ingress for grafana to make the interface accessible from the outside and set an admin password.

Add a values.yaml named /syseleven-kube-prometheus-stack/values-kube-prometheus-stack-development.yaml:

grafana:
  adminPassword: highly-secure-production-password
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-production"
    hosts:
      - grafana.example.com
    tls:
      - secretName: grafana.example.com-tls
        hosts:
          - grafana.example.com
  grafana.ini:
    server:
      root_url: https://grafana.example.com

This way we set our admin password for Grafana, which can be disabled later on. Furthermore, we define the domain where Grafana will be reachable. Nothing more to do! Our reverse proxy stack takes the rest of the missing jobs for it.
Change your global .gitlab-ci.yml to include your observability stack:

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
    - /syseleven-loki-promtail/.gitlab-ci.yml

All we need to do is push it again to our control repository:

#make sure you're inside ~/git/my-project
git add .
git commit -m "add obersavibility stack"
git push

And the next pipeline will be triggered on https://code.syseleven.de/$customer/my-project/-/pipelines!
As soon as the pipeline is finished you're even able to visit your very own Grafana!

Deploy Redis and Percona XtraDB (MySQL)

Last, but not least, we will add Redis and a MySQL derivate called Percona XtraDB cluster to our Kubernetes cluster. This way we have two different databases we can test our own software on, if needed. Redis is an in-memory database, which can be used for caching and fast retrievable data sections, the Percona XtraDB cluster offers you a database management system to work with.

First we add like before a value.yaml at syseleven-redis-ha/values-redis-ha.yaml and set the resource limits to handle the incoming workload in Redis properly:

redis:
  config:
    maxmemory: 1024MB
  resources:
    limits:
      memory: 1280Mi

Then we add some configuration to syseleven-pxc-operator/values-pxc-db.yaml, e.g. as we don't want to enable scheduled backups, so we set:

backup:
  schedule: []

We also want some specific configuration for our database. This time we also use Ansible scripts as these are specific configuration steps, which are not supported in helm charts themselves. The structure changes slightly. We add syseleven-pxc-operator/values-pxc-db-development.yaml:

pxc:
  persistence:
    storageClass: sys11-quobyte-external-provisioner
    size: 10Gi

  resources:
    requests:
      cpu: 300m

haproxy:
  resources:
    requests:
      cpu: 300m

secrets:
  passwords:
    root: $PASSWORD
    xtrabackup: $PASSWORD
    monitor: $PASSWORD
    clustercheck: $PASSWORD
    proxyadmin: $PASSWORD
    pmmserver: $PASSWORD
    operator: $PASSWORD

To create a database and user entry we also need to create an additional file, the syseleven-pxc-operator/values-mysql-config-development.yaml:

databases:
  - name: forms
    state: present
  - name: web
    encoding: utf8mb4
    state: present
users:
  - name: 'web'
    host: '%'
    password: test
    priv: 'web.*:ALL'
    state: present

For more secure ways to add passwords, we recommend password hashing with Ansible.

The Building Blocks will deploy with the main values.yaml file and the environment specific values-$ENV.yaml file combined. This way you are able to overwrite already set configuration in your main-value-file.

Now add those two Building Blocks to your global .gitlab-ci.yml:

stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
    - /syseleven-loki-promtail/.gitlab-ci.yml
    - /syseleven-redis-ha/.gitlab-ci.yml
    - /syseleven-pxc-operator/.gitlab-ci.yml

and start your final pipeline:

#make sure you're inside ~/git/my-project
git add .
git commit -m "add database and cache stacks"
git push

Now your MetaKube cluster is ready to go! You can use the described way to manage your own software stacks. After you finished the development environment, you can replicate all this for the staging and production environments.