Loki-Promtail

The source code and default configuration of the Building Block is available in our GitLab.

Adding the Building Block

Add the directory syseleven-loki-promtail to your control repository. Add a .gitlab-ci.yml to the directory with the following content:

include:
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobDevelopment.yaml
    ref: 3.5.2
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobStaging.yaml
    ref: 3.5.2
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobProduction.yaml
    ref: 3.5.2

Remove environments you are not using by removing their include.

Required configuration

No confguration is required.

What logs are collected?

  • The building block consists of two parts: Loki and Promtail. Promtail runs as a DaemonSet on each node in the cluster and collects logs. Loki is running as a central instance in the cluster and stores the logs it receives from Promtail. When looking into the logs you usually only interact with Loki.
  • The building block collects stdout and stderr of all pods in the cluster. In addition to the pods deployed by the SysEleven provided building blocks, this also includes the logs of all application pods deployed by the user.
  • In addition to the pod logs the systemd journals of all nodes in the cluster are collected.
  • By default the logs are stored by Loki for one week. The retention period can be adjusted in values-loki.yaml see.

How can I access the logs?

The easiest way to access the logs is the Grafana that is deployed with the kube-stack-prometheus building block. We automatically create a Loki datasource in it for you, so you can skip that step in the linked upstream documentation. In Grafana you can use the explore feature to take a look at your logs and refine your search query on the fly. For something more permanent you can add a log panel to one of your dashboards.

If you prefer the command line you can use logcli.

Monitoring

Additional alertrules

  • None

Additional Grafana dashboards

  • Loki Top 10 producer
    • An overview of the top 10 kubernetes namespaces that produce logs
  • Loki & Promtail
    • An overview of performance metrics from loki and promtail

Scale loki volume

If you need to scale the persistent volume Loki stores your data on, perform the following steps:

# Delete StatefulSet (the PVC will remain)
kubectl delete sts loki

# Patch the PVC to e.g. 10Gi
kubectl patch pvc storage-loki-0 -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}' --type=merge

Then, adapt your adapt values.yaml, e.g. values-loki-stage.yaml for the same size as the patch above.

To deploy the StatefulSet again, push your changes and merge them to the default branch - the CI will then deploy it again.

Scale Setup

This building block consists of multiple components. Each of the components can and must be scaled individually.

Scaling Loki

Also see Scaling with Loki for upstream documentation.

  • Scaling replicas is not supported with this building block. See loki-distributed for a possible solution.
  • Requests/limits for CPU/memory can be adjusted

Scaling Promtail

  • Runs once DaemonSet on each node, so no further replica scaling needed
  • Requests/limits for CPU/memory can be adjusted