Deploy a development setup

This tutorial will guide you through a fully functional Kubernetes cluster without your team
doing any global configuration steps during their own application development.
This includes plugins and additional components like:

  1. Ingress controller (ingress-nginx)
  2. External-dns manages our DNS entries (external-dns)
  3. TLS certificate management (cert-manager)
  4. Monitoring (Prometheus with Grafana or Loki-Promtail)
  5. A database (pxc-operator)
  6. General purpose in-memory database (redis-ha)

At the end of this tutorial you have

  • a running Kubernetes cluster
  • installed all necessary Building Blocks for your developers to start immediately
  • have an example ingress defined for kube-prometheus(for the Grafana view)
  • only need to focus on application definition and development while all underlining tasks are either configured or easy to use

Predefinitions and Preparations

To finish this tutorial successfully you need to have a few local tools preinstalled, we mentioned on Getting Started.

What you need

Besides the tools you also need a DNS provider with API-Endpoints, e.g. SysEleven DNS or other providers.
Specifically, you need a DNS provider which is supported by external-dns to manage your subdomain records.

Configure GitLab Settings

Make sure you are familiar with the basic concept by doing our Getting Started tutorial.

First create a MetaKube Cluster. You need at least three nodes of the flavor m1c.medium. Download the kubeconfig and copy it to the path according to your kubectl installation (e.g. ~/.kube/config).
After this login to code.syseleven.de. To centralize all reoccurring tasks we recommend using one repository for CI/CD purposes.

Create a git repository defined as $customer/$project, e.g. "syseleven/my-project". We will refer to this repository as control repository, as we control all necessary tasks regarding the setup on MetaKube from here out.
Adjust the settings as described in setup a control repository and add your kubeconfig as CI/CD variable accordingly. Our variable KUBECONFIG will be assigned to the environment "development".

We recommend the mkactl tool, to setup your development environment.

Start Structuring your Control Repository

Do these steps only if you are not using the mkactl tool.

Clone your control repository locally. We assume you clone the repository onto the local path ~/git/my-project.:

cd ~/git
export customer=set-your-namespace
git clone git@code.syseleven.de:$customer/my-project.git

We will work on the main branch for now. After this tutorial you can change the environment settings to adjust your branches as you need to.
The directory structure will be as follows:

/
└── syseleven-kube-prometheus-stack/
    ├── .gitlab-ci.yml
    ├── values-kube-prometheus-stack-development.yaml
└── syseleven-loki-promtail/
    ├── .gitlab-ci.yml
└── syseleven-ingress/
    ├── .gitlab-ci.yml
└── syseleven-external-dns/
    ├── .gitlab-ci.yml
    └── values-external-dns.yaml
└── syseleven-cert-manager/
    ├── .gitlab-ci.yml
└── syseleven-redis-ha/
    ├── .gitlab-ci.yml
    └── values-redis-ha.yaml
└── syseleven-pxc-operator/
    ├── .gitlab-ci.yml
    ├── values-pxc-db-development.yaml
    └── values-pxc-db.yaml
.gitlab-ci.yml

use mkactl

The mkactl tool offers you a less error prone way to set up your development environment.
If you install and setup the mkactl correctly mkactl, you can proceed with the following steps.

mkactl block add ingress-nginx
mkactl block add cert-manager
mkactl block add kube-prometheus-stack
mkactl block add loki-promtail
mkactl block add pxc-operator
mkactl block add redis-ha

Wait until each pipeline succeeded.

Do a git pull and your local workspace should look like:

drwxr-xr-x   4 marcus  staff   128 Jun 16 11:34 ..
-rw-r--r--   1 marcus  staff  6261 Jun 16 11:35 README.md
-rwxr-xr-x   1 marcus  staff   214 Jun 16 12:12 .gitlab-ci.yml
drwxr-xr-x   6 marcus  staff   192 Jun 16 12:12 cert-manager
drwxr-xr-x   6 marcus  staff   192 Jun 16 12:12 ingress-nginx
drwxr-xr-x   6 marcus  staff   192 Jun 16 12:12 kube-prometheus-stack
drwxr-xr-x   6 marcus  staff   192 Jun 16 12:12 loki-promtail
drwxr-xr-x  10 marcus  staff   320 Jun 16 12:12 .
drwxr-xr-x   6 marcus  staff   192 Jun 16 12:12 pxc-operator
drwxr-xr-x   6 marcus  staff   192 Jun 17 15:53 redis-ha
drwxr-xr-x  14 marcus  staff   448 Jun 16 12:12 .git

Check the deployment with:

marcus@marcus mka-497 % kubectl get ns
NAME                              STATUS   AGE
cloud-init-settings               Active   28h
default                           Active   28h
kube-node-lease                   Active   28h
kube-public                       Active   28h
kube-system                       Active   28h
syseleven-cert-manager            Active   27h
syseleven-ingress-nginx           Active   27h
syseleven-kube-prometheus-stack   Active   27h
syseleven-loki-promtail           Active   27h
syseleven-pxc-operator            Active   27h
syseleven-redis-ha                Active   24m

On the root path lies a .gitlab-ci.yml and a subfolder for each Building Block. This ensures a better reading structure. Each subfolder expects its own .gitlab-ci.yml where the repository of the corresponding Building Block will be included. An example file follows underneath:

The mkactl command line tool will generate them for you.

include:
  - project: syseleven/building-blocks/helmfiles/cert-manager
    file: JobDevelopment.yaml
    ref: 4.31.1

The version ref will be the latest from our curated building blocks.

The project refers to the repository residing on [code.syseleven.de]. The file defines the environment, which can be development, staging or production. The ref defines the appropriate branch at which the specific version can be referenced to. The main branch is hereby the latest branch.

The global pipeline definition will be written in the main directory also as .gitlab-ci.yml and needs two stages defined: diff and deploy. Those are necessary for the Building Blocks, as they define jobs in those two stages.
See further in the script below on how to write it in your .gitlab-ci.yml.
For each Building Block you can look at our Building Block Catalogue to see, what to add to the Building-Block-specific .gitlab-ci.yaml and values.yaml for each one in the subfolders.

Enable Building Blocks Step by Step

Deploy the Stack for Service Proxying

After setting up the folder structure we will now enable the Building Blocks incrementally.

At first, we start with the Kubernetes reverse proxy stack. This includes:

  1. ingress-nginx
  2. cert-manager
  3. external-dns

This manages our DNS entries for our domain (external-dns), the certificates via Let's Encrypt (cert-manager) and gives our cluster the ability to create and manage ingress definitions (ingress-nginx).

The ingress-controller and cert-manager does not require configuration. The external-dns building block requires credentials to access your DNS providers API. For a list of supported DNS providers referto (addlink).

For Syseleven DNS Zones,use the following configuration snippet. I'd consider adding AWS or similar to the list of provided configuration snippets.
What happens if I don't have access to a DNS providers API (common problem in larger organizations)- I should still be able to continue. Wildcard subdomains are an option.

designate:
  username: "$sysElevenStackUser"
  password: "$sysElevenStackPassword"
  projectName: "the SysEleven Stack project name"

After you saved this file, add the Building Blocks to your global pipeline configuration, which will be written in .gitlab-ci.yml. It should now look like this:

stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml

Now it is set and done. Push the changes to your control repository

#make sure you're inside ~/git/my-project
git add .
git commit -m "add reverse proxy BBs"
git push

and watch the pipeline work on https://code.syseleven.de/$customer/my-project/-/pipelines!

The Ingress Controller needs some time to allocate a floating IP for its service. Check it with kubectl get svc -n syseleven-ingress-nginx -wand wait for the pending state to change to a valid IP!

Deploy Obersavibility Stack with an Ingress Resource for Grafana

After the pipeline finished, we will start to add the observability stack.
This includes tools like Prometheus for metrics, Loki/Promtail for logs, the Alertmanager for alerts and Grafana as GUI. We will also set an ingress for Grafana to be able to start right off seeing all necessary metrics.

For this to work we need to adjust the tooling a bit more. The kube-prometheus stack is highly customizable and would need configuration changes in a production environment, for example alerting rules. Both Loki and promtail rarely require customization. You can find all customization options in the description of the Building Block. For this tutorial, we only need to configure an ingress for grafana to make the interface accessible from the outside and set an admin password.

Add a values.yaml named /syseleven-kube-prometheus-stack/values-kube-prometheus-stack-development.yaml:

grafana:
  adminPassword: highly-secure-production-password
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-production"
    hosts:
      - grafana.example.com
    tls:
      - secretName: grafana.example.com-tls
        hosts:
          - grafana.example.com
  grafana.ini:
    server:
      root_url: https://grafana.example.com

This way we set our admin password for Grafana, which can be disabled later on. Furthermore, we define the domain where Grafana will be reachable. Nothing more to do! Our reverse proxy stack takes the rest of the missing jobs for it.
Change your global .gitlab-ci.yml to include your observability stack:

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
    - /syseleven-loki-promtail/.gitlab-ci.yml

All we need to do is push it again to our control repository:

#make sure you're inside ~/git/my-project
git add .
git commit -m "add obersavibility stack"
git push

And the next pipeline will be triggered on https://code.syseleven.de/$customer/my-project/-/pipelines!
As soon as the pipeline is finished you're even able to visit your very own Grafana!

Deploy Redis and Percona XtraDB (MySQL)

Last, but not least, we will add Redis, and a MySQL derivate called Percona XtraDB cluster to our Kubernetes cluster. This way we have two different databases we can test our own software on, if needed. Redis is an in-memory database, which can be used for caching and fast retrievable data sections, the Percona XtraDB cluster offers you a database management system to work with.

First we add like before a value.yaml at syseleven-redis-ha/values-redis-ha.yaml and set the resource limits to handle the incoming workload in Redis properly:

redis:
  config:
    maxmemory: 1024MB
  resources:
    limits:
      memory: 1280Mi

Then we add some configuration to syseleven-pxc-operator/values-pxc-db.yaml, e.g. as we don't want to enable scheduled backups, so we set:

backup:
  schedule: []

We also want some specific configuration for our database. This time we also use Ansible scripts as these are specific configuration steps, which are not supported in helm charts themselves. The structure changes slightly. We add syseleven-pxc-operator/values-pxc-db-development.yaml:

pxc:
  persistence:
    storageClass: sys11-quobyte-external-provisioner
    size: 10Gi

  resources:
    requests:
      cpu: 300m

haproxy:
  resources:
    requests:
      cpu: 300m

secrets:
  passwords:
    root: $PASSWORD
    xtrabackup: $PASSWORD
    monitor: $PASSWORD
    clustercheck: $PASSWORD
    proxyadmin: $PASSWORD
    pmmserver: $PASSWORD
    operator: $PASSWORD

To create a database and user entry we also need to create an additional file, the syseleven-pxc-operator/values-mysql-config-development.yaml:

databases:
  - name: forms
    state: present
  - name: web
    encoding: utf8mb4
    state: present
users:
  - name: 'web'
    host: '%'
    password: test
    priv: 'web.*:ALL'
    state: present

For more secure ways to add passwords, we recommend password hashing with Ansible.

The Building Blocks will deploy with the main values.yaml file, and the environment specific values-$ENV.yaml file combined. This way you are able to overwrite already set configuration in your main-value-file.

Now add those two Building Blocks to your global .gitlab-ci.yml:

stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
    - /syseleven-loki-promtail/.gitlab-ci.yml
    - /syseleven-redis-ha/.gitlab-ci.yml
    - /syseleven-pxc-operator/.gitlab-ci.yml

and start your final pipeline:

#make sure you're inside ~/git/my-project
git add .
git commit -m "add database and cache stacks"
git push

Now your MetaKube cluster is ready to go! You can use the described way to manage your own software stacks. After you finished the development environment, you can replicate all this for the staging and production environments.