Loki-Promtail

Overview

The source code and default configuration of the Building Block is available in our code.sysEleven.de. Infos on release notes and new features please follow Release notes loki-promtail

Prerequisites on loki promtail

Proceed with the following prerequisite description to use the Building Block out of the box.

Recommended setup

A recommended resource overview is listed in the table below.

No further activities need to be carried out in advance.

Adding the Building Block

Add the directory syseleven-loki-promtail to your control repository. Add a .gitlab-ci.yml to the directory with the following content:

include:
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobDevelopment.yaml
    ref: 11.10.0
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobStaging.yaml
    ref: 11.10.0
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobProduction.yaml
    ref: 11.10.0

Required configuration

No confguration is required.

What logs are collected?

• The building block consists of two parts: Loki and Promtail. Promtail runs as a DaemonSet on each node in the cluster and collects logs. Loki is running as a central instance in the cluster and stores the logs it receives from Promtail. When looking into the logs, you usually only interact with Loki.
• The building block collects stdout and stderr of all pods in the cluster. In addition to the pods deployed by the SysEleven provided building blocks, this also includes the logs of all application pods deployed by the user.
• In addition to the pod logs the systemd journals of all nodes in the cluster are collected.
• By default the logs are stored by Loki for one week. The retention period can be adjusted in values-loki.yaml see.

How can I access the logs?

The easiest way to access the logs is Grafana provided with the kube-stack-prometheus module. We automatically create a Loki data source for you in it, so you can skip this step in the linked upstream documentation. In Grafana, you can use the explore feature to take a look at your logs and refine your query.
For something more permanent, you can add a log panel to one of your dashboards.

If you prefer the command line you can use logcli.

Monitoring

Additional Alertrules

• None

Additional Grafana dashboards

• Loki Top 10 producer
  • An overview of the top 10 kubernetes namespaces that produce logs
• Loki & Promtail
  • An overview of performance metrics from Loki and Promtail

Scale loki volume

If you need to scale the persistent volume Loki stores your data on, perform the following steps:

# Delete StatefulSet (the PVC will remain)
kubectl delete sts loki

# Patch the PVC to e.g. 10Gi
kubectl patch pvc storage-loki-0 -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}' --type=merge

Then, adapt your adapt values.yaml, e.g. values-loki-stage.yaml for the same size as the patch above.

To deploy the StatefulSet again, push your changes and merge them to the default branch - the CI will then deploy it again.

Scale Setup

This building block consists of multiple components. Each of the components can and must be scaled individually.

Scaling Loki

Also see Scaling with Loki for upstream documentation.

• Scaling replicas is not supported with this building block. See loki-distributed for a possible solution.
• Requests/limits for CPU/memory can be adjusted

Scaling Promtail

• Runs as one DaemonSet on each node, so no further replica scaling is needed
• Requests/limits for CPU/memory can be adjusted

Migrate from Loki-Promtail to Loki-Alloy

Prepare Setup

Delete the clusterrole and clusterrolebinding for loki in your cluster:

kubectl delete clusterrole loki-clusterrole
kubectl delete clusterrolebinding loki-clusterrolebinding
kubectl delete crd podlogs.monitoring.grafana.com
helm uninstall -n syseleven-loki-promtail promtail

Install Loki-Alloy

If you have a control-repository with a .gitlab-ci.yml, adjust the package you install:

stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
#    - /syseleven-loki-promtail/.gitlab-ci.yml # or delete entirely
    - /syseleven-loki-alloy/.gitlab-ci.yml

And add a directory called syseleven-loki-alloy with another .gitlab-ci.yml:

include:
  # Development environment
  - project: syseleven/building-blocks/helmfiles/loki-alloy
    file: JobDevelopment.yaml
    ref: main

Go to your grafana domain, login and go to Connections > Data Sources. Click the blue button Add new data source. Choose under Logging & document databases loki as source.

Under Connection add the url for the new loki instance. The Default is http://loki.syseleven-loki-alloy:3100

Cleanup

After the end of your retention period (which defaults to 7 days) you can uninstall loki-promtail. To do so, make sure you got the CLI helm installed. Afterwards it is safe to remove the namespace.

helm uninstall -n syseleven-loki-promtail loki loki-promtail-extension event-exporter promtail
kubectl delete namespace syseleven-loki-promtail

Release-Notes

Please find more infos on release notes and new features Release notes Loki-Promtail