This tutorial will guide you through a fully functional Kubernetes cluster without your team
doing any global configuration steps during their own application development.
This includes plugins and additional components like:
At the end of this tutorial you have
To finish this tutorial successfully you need to have a few local tools preinstalled, we mentioned on Getting Started.
Besides the tools you also need a DNS provider with API-Endpoints, e.g. SysEleven DNS or other providers.
Specifically, you need a DNS provider which is supported by external-dns to manage your subdomain records.
Make sure you are familiar with the basic concept by doing our Getting Started tutorial.
First create a MetaKube Cluster. You need at least three nodes of the flavor m1c.medium. Download the kubeconfig and copy it to the path according to your kubectl installation (e.g.
After this login to code.syseleven.de. To centralize all reoccurring tasks we recommend using one repository for CI/CD purposes.
Create a git repository defined as
$customer/$project, e.g. "syseleven/my-project". We will refer to this repository as control repository, as we control all necessary tasks regarding the setup on MetaKube from here out.
Adjust the settings as described in setup a control repository and add your kubeconfig as CI/CD variable accordingly. Our variable KUBECONFIG will be assigned to the environment "development".
We recommend the mkactl tool, to setup your development environment.
Do these steps only if you are not using the mkactl tool.
Clone your control repository locally. We assume you clone the repository onto the local path
cd ~/git export customer=set-your-namespace git clone firstname.lastname@example.org:$customer/my-project.git
We will work on the main branch for now. After this tutorial you can change the environment settings to adjust your branches as you need to.
The directory structure will be as follows:
/ └── syseleven-kube-prometheus-stack/ ├── .gitlab-ci.yml ├── values-kube-prometheus-stack-development.yaml └── syseleven-loki-promtail/ ├── .gitlab-ci.yml └── syseleven-ingress/ ├── .gitlab-ci.yml └── syseleven-external-dns/ ├── .gitlab-ci.yml └── values-external-dns.yaml └── syseleven-cert-manager/ ├── .gitlab-ci.yml └── syseleven-redis-ha/ ├── .gitlab-ci.yml └── values-redis-ha.yaml └── syseleven-pxc-operator/ ├── .gitlab-ci.yml ├── values-pxc-db-development.yaml └── values-pxc-db.yaml .gitlab-ci.yml
The mkactl tool offers you a less error prone way to set up your development environment.
If you install and setup the mkactl correctly mkactl, you can proceed with the following steps.
mkactl block add ingress-nginx mkactl block add cert-manager mkactl block add kube-prometheus-stack mkactl block add loki-promtail mkactl block add pxc-operator mkactl block add redis-ha
Wait until each pipeline succeeded.
Do a git pull and your local workspace should look like:
drwxr-xr-x 4 marcus staff 128 Jun 16 11:34 .. -rw-r--r-- 1 marcus staff 6261 Jun 16 11:35 README.md -rwxr-xr-x 1 marcus staff 214 Jun 16 12:12 .gitlab-ci.yml drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 cert-manager drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 ingress-nginx drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 kube-prometheus-stack drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 loki-promtail drwxr-xr-x 10 marcus staff 320 Jun 16 12:12 . drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 pxc-operator drwxr-xr-x 6 marcus staff 192 Jun 17 15:53 redis-ha drwxr-xr-x 14 marcus staff 448 Jun 16 12:12 .git
Check the deployment with:
marcus@marcus mka-497 % kubectl get ns NAME STATUS AGE cloud-init-settings Active 28h default Active 28h kube-node-lease Active 28h kube-public Active 28h kube-system Active 28h syseleven-cert-manager Active 27h syseleven-ingress-nginx Active 27h syseleven-kube-prometheus-stack Active 27h syseleven-loki-promtail Active 27h syseleven-pxc-operator Active 27h syseleven-redis-ha Active 24m
On the root path lies a
.gitlab-ci.yml and a subfolder for each Building Block. This ensures a better reading structure. Each subfolder expects its own
.gitlab-ci.yml where the repository of the corresponding Building Block will be included. An example file follows underneath:
The mkactl command line tool will generate them for you.
include: - project: syseleven/building-blocks/helmfiles/cert-manager file: JobDevelopment.yaml ref: 4.35.0
The version ref will be the latest from our curated building blocks.
The project refers to the repository residing on [code.syseleven.de]. The file defines the environment, which can be development, staging or production. The ref defines the appropriate branch at which the specific version can be referenced to. The
main branch is hereby the
The global pipeline definition will be written in the main directory also as
.gitlab-ci.yml and needs two stages defined: diff and deploy. Those are necessary for the Building Blocks, as they define jobs in those two stages.
See further in the script below on how to write it in your .gitlab-ci.yml.
For each Building Block you can look at our Building Block Catalogue to see, what to add to the Building-Block-specific .gitlab-ci.yaml and values.yaml for each one in the subfolders.
After setting up the folder structure we will now enable the Building Blocks incrementally.
At first, we start with the Kubernetes reverse proxy stack. This includes:
This manages our DNS entries for our domain (external-dns), the certificates via Let's Encrypt (cert-manager) and gives our cluster the ability to create and manage ingress definitions (ingress-nginx).
The ingress-controller and cert-manager does not require configuration. The external-dns building block requires credentials to access your DNS providers API. For a list of supported DNS providers referto (addlink).
For Syseleven DNS Zones,use the following configuration snippet. I'd consider adding AWS or similar to the list of provided configuration snippets.
What happens if I don't have access to a DNS providers API (common problem in larger organizations)- I should still be able to continue. Wildcard subdomains are an option.
designate: username: "$sysElevenStackUser" password: "$sysElevenStackPassword" projectName: "the SysEleven Stack project name"
After you saved this file, add the Building Blocks to your global pipeline configuration, which will be written in
.gitlab-ci.yml. It should now look like this:
stages: - diff - deploy include: - /syseleven-ingress-nginx/.gitlab-ci.yml - /syseleven-cert-manager/.gitlab-ci.yml - /syseleven-external-dns/.gitlab-ci.yml
Now it is set and done. Push the changes to your control repository
#make sure you're inside ~/git/my-project git add . git commit -m "add reverse proxy BBs" git push
and watch the pipeline work on
The Ingress Controller needs some time to allocate a floating IP for its service. Check it with
kubectl get svc -n syseleven-ingress-nginx -wand wait for the pending state to change to a valid IP!
After the pipeline finished, we will start to add the observability stack.
This includes tools like Prometheus for metrics, Loki/Promtail for logs, the Alertmanager for alerts and Grafana as GUI. We will also set an ingress for Grafana to be able to start right off seeing all necessary metrics.
For this to work we need to adjust the tooling a bit more. The kube-prometheus stack is highly customizable and would need configuration changes in a production environment, for example alerting rules. Both Loki and promtail rarely require customization. You can find all customization options in the description of the Building Block. For this tutorial, we only need to configure an ingress for grafana to make the interface accessible from the outside and set an admin password.
Add a values.yaml named
grafana: adminPassword: highly-secure-production-password ingress: enabled: true annotations: cert-manager.io/cluster-issuer: "letsencrypt-production" hosts: - grafana.example.com tls: - secretName: grafana.example.com-tls hosts: - grafana.example.com grafana.ini: server: root_url: https://grafana.example.com
This way we set our admin password for Grafana, which can be disabled later on. Furthermore, we define the domain where Grafana will be reachable. Nothing more to do! Our reverse proxy stack takes the rest of the missing jobs for it.
Change your global
.gitlab-ci.yml to include your observability stack:
include: - /syseleven-ingress-nginx/.gitlab-ci.yml - /syseleven-cert-manager/.gitlab-ci.yml - /syseleven-external-dns/.gitlab-ci.yml - /syseleven-kube-prometheus-stack/.gitlab-ci.yml - /syseleven-loki-promtail/.gitlab-ci.yml
All we need to do is push it again to our control repository:
#make sure you're inside ~/git/my-project git add . git commit -m "add obersavibility stack" git push
And the next pipeline will be triggered on
As soon as the pipeline is finished you're even able to visit your very own Grafana!
Last, but not least, we will add Redis, and a MySQL derivate called Percona XtraDB cluster to our Kubernetes cluster. This way we have two different databases we can test our own software on, if needed. Redis is an in-memory database, which can be used for caching and fast retrievable data sections, the Percona XtraDB cluster offers you a database management system to work with.
First we add like before a value.yaml at
syseleven-redis-ha/values-redis-ha.yaml and set the resource limits to handle the incoming workload in Redis properly:
redis: config: maxmemory: 1024MB resources: limits: memory: 1280Mi
Then we add some configuration to
syseleven-pxc-operator/values-pxc-db.yaml, e.g. as we don't want to enable scheduled backups, so we set:
backup: schedule: 
We also want some specific configuration for our database. This time we also use Ansible scripts as these are specific configuration steps, which are not supported in helm charts themselves. The structure changes slightly. We add
pxc: persistence: storageClass: sys11-quobyte-external-provisioner size: 10Gi resources: requests: cpu: 300m haproxy: resources: requests: cpu: 300m secrets: passwords: root: $PASSWORD xtrabackup: $PASSWORD monitor: $PASSWORD clustercheck: $PASSWORD proxyadmin: $PASSWORD pmmserver: $PASSWORD operator: $PASSWORD
To create a database and user entry we also need to create an additional file, the
databases: - name: forms state: present - name: web encoding: utf8mb4 state: present users: - name: 'web' host: '%' password: test priv: 'web.*:ALL' state: present
For more secure ways to add passwords, we recommend password hashing with Ansible.
The Building Blocks will deploy with the main values.yaml file, and the environment specific values-$ENV.yaml file combined. This way you are able to overwrite already set configuration in your main-value-file.
Now add those two Building Blocks to your global
stages: - diff - deploy include: - /syseleven-ingress-nginx/.gitlab-ci.yml - /syseleven-cert-manager/.gitlab-ci.yml - /syseleven-external-dns/.gitlab-ci.yml - /syseleven-kube-prometheus-stack/.gitlab-ci.yml - /syseleven-loki-promtail/.gitlab-ci.yml - /syseleven-redis-ha/.gitlab-ci.yml - /syseleven-pxc-operator/.gitlab-ci.yml
and start your final pipeline:
#make sure you're inside ~/git/my-project git add . git commit -m "add database and cache stacks" git push
Now your MetaKube cluster is ready to go! You can use the described way to manage your own software stacks. After you finished the development environment, you can replicate all this for the staging and production environments.