This tutorial will guide you through a fully functional Kubernetes cluster without your team
doing any global configuration steps during their own application development.
This includes plugins and additional components like:
At the end of this tutorial you have
To finish this tutorial successfully you need to have a few local tools preinstalled, we mentioned on Getting Started.
Besides the tools you also need a DNS provider with API-Endpoints, e.g. SysEleven DNS or other providers.
Specifically, you need a DNS provider which is supported by external-dns to manage your subdomain records.
Make sure you are familiar with the basic concept by doing our Getting Started tutorial.
First create a MetaKube Cluster. You need at least three nodes of the flavor m1c.medium. Download the kubeconfig and copy it to the path according to your kubectl installation (e.g. ~/.kube/config).
After this login to code.syseleven.de. To centralize all reoccurring tasks we recommend using one repository for CI/CD purposes.
Create a git repository defined as $customer/$project, e.g. "syseleven/my-project". We will refer to this repository as control repository, as we control all necessary tasks regarding the setup on MetaKube from here out.
Adjust the settings as described in setup a control repository and add your kubeconfig as CI/CD variable accordingly. Our variable KUBECONFIG will be assigned to the environment "development".
We recommend the mkactl tool, to setup your development environment.
Do these steps only if you are not using the mkactl tool.
Clone your control repository locally. We assume you clone the repository onto the local path ~/git/my-project.:
cd ~/git
export customer=set-your-namespace
git clone git@code.syseleven.de:$customer/my-project.gitWe will work on the main branch for now. After this tutorial you can change the environment settings to adjust your branches as you need to.
The directory structure will be as follows:
/
└── syseleven-kube-prometheus-stack/
├── .gitlab-ci.yml
├── values-kube-prometheus-stack-development.yaml
└── syseleven-loki-promtail/
├── .gitlab-ci.yml
└── syseleven-ingress/
├── .gitlab-ci.yml
└── syseleven-external-dns/
├── .gitlab-ci.yml
└── values-external-dns.yaml
└── syseleven-cert-manager/
├── .gitlab-ci.yml
└── syseleven-redis-ha/
├── .gitlab-ci.yml
└── values-redis-ha.yaml
└── syseleven-pxc-operator/
├── .gitlab-ci.yml
├── values-pxc-db-development.yaml
└── values-pxc-db.yaml
.gitlab-ci.ymlThe mkactl tool offers you a less error prone way to set up your development environment.
If you install and setup the mkactl correctly mkactl, you can proceed with the following steps.
mkactl block add ingress-nginx
mkactl block add cert-manager
mkactl block add kube-prometheus-stack
mkactl block add loki-promtail
mkactl block add pxc-operator
mkactl block add redis-haWait until each pipeline succeeded.
Do a git pull and your local workspace should look like:
drwxr-xr-x 4 marcus staff 128 Jun 16 11:34 ..
-rw-r--r-- 1 marcus staff 6261 Jun 16 11:35 README.md
-rwxr-xr-x 1 marcus staff 214 Jun 16 12:12 .gitlab-ci.yml
drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 cert-manager
drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 ingress-nginx
drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 kube-prometheus-stack
drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 loki-promtail
drwxr-xr-x 10 marcus staff 320 Jun 16 12:12 .
drwxr-xr-x 6 marcus staff 192 Jun 16 12:12 pxc-operator
drwxr-xr-x 6 marcus staff 192 Jun 17 15:53 redis-ha
drwxr-xr-x 14 marcus staff 448 Jun 16 12:12 .gitCheck the deployment with:
marcus@marcus mka-497 % kubectl get ns
NAME STATUS AGE
cloud-init-settings Active 28h
default Active 28h
kube-node-lease Active 28h
kube-public Active 28h
kube-system Active 28h
syseleven-cert-manager Active 27h
syseleven-ingress-nginx Active 27h
syseleven-kube-prometheus-stack Active 27h
syseleven-loki-promtail Active 27h
syseleven-pxc-operator Active 27h
syseleven-redis-ha Active 24mOn the root path lies a .gitlab-ci.yml and a subfolder for each Building Block. This ensures a better reading structure. Each subfolder expects its own .gitlab-ci.yml where the repository of the corresponding Building Block will be included. An example file follows underneath:
The mkactl command line tool will generate them for you.
include:
- project: syseleven/building-blocks/helmfiles/cert-manager
file: JobDevelopment.yaml
ref: 9.1.0The version ref will be the latest from our curated building blocks.
The project refers to the repository residing on [code.syseleven.de]. The file defines the environment, which can be development, staging or production. The ref defines the appropriate branch at which the specific version can be referenced to. The main branch is hereby the latest branch.
The global pipeline definition will be written in the main directory also as .gitlab-ci.yml and needs two stages defined: diff and deploy. Those are necessary for the Building Blocks, as they define jobs in those two stages.
See further in the script below on how to write it in your .gitlab-ci.yml.
For each Building Block you can look at our Building Block Catalogue to see, what to add to the Building-Block-specific .gitlab-ci.yaml and values.yaml for each one in the subfolders.
After setting up the folder structure we will now enable the Building Blocks incrementally.
At first, we start with the Kubernetes reverse proxy stack. This includes:
This manages our DNS entries for our domain (external-dns), the certificates via Let's Encrypt (cert-manager) and gives our cluster the ability to create and manage ingress definitions (ingress-nginx).
The ingress-controller and cert-manager does not require configuration. The external-dns building block requires credentials to access your DNS providers API. For a list of supported DNS providers referto (addlink).
For Syseleven DNS Zones,use the following configuration snippet. I'd consider adding AWS or similar to the list of provided configuration snippets.
What happens if I don't have access to a DNS providers API (common problem in larger organizations)- I should still be able to continue. Wildcard subdomains are an option.
designate:
username: "$sysElevenStackUser"
password: "$sysElevenStackPassword"
projectName: "the SysEleven Stack project name"After you saved this file, add the Building Blocks to your global pipeline configuration, which will be written in .gitlab-ci.yml. It should now look like this:
stages:
- diff
- deploy
include:
- /syseleven-ingress-nginx/.gitlab-ci.yml
- /syseleven-cert-manager/.gitlab-ci.yml
- /syseleven-external-dns/.gitlab-ci.ymlNow it is set and done. Push the changes to your control repository
#make sure you're inside ~/git/my-project
git add .
git commit -m "add reverse proxy BBs"
git pushand watch the pipeline work on https://code.syseleven.de/$customer/my-project/-/pipelines!
The Ingress Controller needs some time to allocate a floating IP for its service. Check it with kubectl get svc -n syseleven-ingress-nginx -wand wait for the pending state to change to a valid IP!
After the pipeline finished, we will start to add the observability stack.
This includes tools like Prometheus for metrics, Loki/Promtail for logs, the Alertmanager for alerts and Grafana as GUI. We will also set an ingress for Grafana to be able to start right off seeing all necessary metrics.
For this to work we need to adjust the tooling a bit more. The kube-prometheus stack is highly customizable and would need configuration changes in a production environment, for example alerting rules. Both Loki and promtail rarely require customization. You can find all customization options in the description of the Building Block. For this tutorial, we only need to configure an ingress for grafana to make the interface accessible from the outside and set an admin password.
Add a values.yaml named /syseleven-kube-prometheus-stack/values-kube-prometheus-stack-development.yaml:
grafana:
adminPassword: highly-secure-production-password
ingress:
enabled: true
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-production"
hosts:
- grafana.example.com
tls:
- secretName: grafana.example.com-tls
hosts:
- grafana.example.com
grafana.ini:
server:
root_url: https://grafana.example.comThis way we set our admin password for Grafana, which can be disabled later on. Furthermore, we define the domain where Grafana will be reachable. Nothing more to do! Our reverse proxy stack takes the rest of the missing jobs for it.
Change your global .gitlab-ci.yml to include your observability stack:
include:
- /syseleven-ingress-nginx/.gitlab-ci.yml
- /syseleven-cert-manager/.gitlab-ci.yml
- /syseleven-external-dns/.gitlab-ci.yml
- /syseleven-kube-prometheus-stack/.gitlab-ci.yml
- /syseleven-loki-promtail/.gitlab-ci.ymlAll we need to do is push it again to our control repository:
#make sure you're inside ~/git/my-project
git add .
git commit -m "add obersavibility stack"
git pushAnd the next pipeline will be triggered on https://code.syseleven.de/$customer/my-project/-/pipelines!
As soon as the pipeline is finished you're even able to visit your very own Grafana!
Last, but not least, we will add Redis, and a MySQL derivate called Percona XtraDB cluster to our Kubernetes cluster. This way we have two different databases we can test our own software on, if needed. Redis is an in-memory database, which can be used for caching and fast retrievable data sections, the Percona XtraDB cluster offers you a database management system to work with.
First we add like before a value.yaml at syseleven-redis-ha/values-redis-ha.yaml and set the resource limits to handle the incoming workload in Redis properly:
redis:
config:
maxmemory: 1024MB
resources:
limits:
memory: 1280MiThen we add some configuration to syseleven-pxc-operator/values-pxc-db.yaml, e.g. as we don't want to enable scheduled backups, so we set:
backup:
schedule: []We also want some specific configuration for our database. This time we also use Ansible scripts as these are specific configuration steps, which are not supported in helm charts themselves. The structure changes slightly. We add syseleven-pxc-operator/values-pxc-db-development.yaml:
pxc:
persistence:
storageClass: sys11-quobyte-external-provisioner
size: 10Gi
resources:
requests:
cpu: 300m
haproxy:
resources:
requests:
cpu: 300m
secrets:
passwords:
root: $PASSWORD
xtrabackup: $PASSWORD
monitor: $PASSWORD
clustercheck: $PASSWORD
proxyadmin: $PASSWORD
pmmserver: $PASSWORD
operator: $PASSWORD
To create a database and user entry we also need to create an additional file, the syseleven-pxc-operator/values-mysql-config-development.yaml:
databases:
- name: forms
state: present
- name: web
encoding: utf8mb4
state: present
users:
- name: 'web'
host: '%'
password: test
priv: 'web.*:ALL'
state: presentFor more secure ways to add passwords, we recommend password hashing with Ansible.
The Building Blocks will deploy with the main values.yaml file, and the environment specific values-$ENV.yaml file combined. This way you are able to overwrite already set configuration in your main-value-file.
Now add those two Building Blocks to your global .gitlab-ci.yml:
stages:
- diff
- deploy
include:
- /syseleven-ingress-nginx/.gitlab-ci.yml
- /syseleven-cert-manager/.gitlab-ci.yml
- /syseleven-external-dns/.gitlab-ci.yml
- /syseleven-kube-prometheus-stack/.gitlab-ci.yml
- /syseleven-loki-promtail/.gitlab-ci.yml
- /syseleven-redis-ha/.gitlab-ci.yml
- /syseleven-pxc-operator/.gitlab-ci.ymland start your final pipeline:
#make sure you're inside ~/git/my-project
git add .
git commit -m "add database and cache stacks"
git pushNow your MetaKube cluster is ready to go! You can use the described way to manage your own software stacks. After you finished the development environment, you can replicate all this for the staging and production environments.