Monitoring is a complex topic and there are many articles, conference talks and books about it. It is impossible to learn everything in a single tutorial, but we want to give you an overview of the monitoring features our Building Blocks already provide for you and offer you some suggestions to additional resources about this topic.
The most important monitoring Building Block is the kube-prometheus-stack which deploys a Prometheus Operator into your cluster, this provides you with a Prometheus server to scrape all metrics, node-exporters to export metrics from all nodes and a Grafana instance to visualize the metrics. The Building Block ships with SysElevens best practices by default and we already include a lot of helpful dashboards for you.
All other Building Blocks are configured to expose their metrics to the central Prometheus instance. When applicable the Building Blocks also deploy additional dashboards to Grafana and alert rules to Prometheus. This way the kube-prometheus-stack Building Block and all other Building Blocks work together to ensure that all services are monitored by default.
In addition to the metrics we also offer the loki-promtail Building Block which deploys a central Loki instance and one Promtail instance on each node. The Promtail instances ship the logs of all pods on the node and the journald log of the node itself to the central Loki instance. The Building Block also adds Loki to the Grafana instance of the kube-prometheus Building Block as a datasource. This way you can access all logs in the Grafana interface.
Both Building Blocks already provide a lot of insights into your cluster. In case you want to add additional alert rules you can follow our example. To also get notified of any issues in the cluster we strongly recommend to configure the alertmanager to forward alerts to you.
All metrics and logs collected by Prometheus and Loki can be visualized by Grafana which is part of the kube-prometheus-stack Building Block. Following our instructions you can expose the Grafana interface via an ingress for a comfortable way to access your dashboards. Speaking about dashboards, it is also possible to add your own dashboards with the Building Block.