How to setup monitoring on Kubernetes using Prometheus, Grafana, AlertManager and Loki
In this document we are using Managed Kubernetes with self-managed nodes. We will learn
how to install Grafana, Prometheus, AlertManager and Loki with persistent volume and
retention period. We will setup dashboard on Grafana and some alerts also for AlertManager.
Pre-requisites:
1. Kubernetes cluster
2. Kubectl
3. Helm
1. How to install Grafana?
We will be using kubectl to deploy Grafana in the Kubernetes cluster.
Deploy.yml:
# kubectl apply -f deploy.yml -n monitoring
Service.yml:
# kubectl apply -f service.yml -n monitoring
Now, our Grafana is ready to use.
# kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/grafana-85f94979d7-ntfjh 1/1 Running 0 4m18s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana-svc NodePort 10.96.241.106 <none> 3000:30011/TCP 7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 4m18s
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-85f94979d7 1 1 1 4m18s
Now, our Grafana is ready. You can access UI using http://<NodeIP>:30011.
2. How to install Prometheus and AlertManager?
Set parameters in prom.yml and alert.yml. Do not install alertmanager in Prometheus(to use its own
alert.yml)
./helm upgrade --install -f prom.yml prometheus prometheus-community/prometheus -n
monitoring
./helm upgrade --install -f alert.yml alertmanager prometheus-community/alertmanager -n
monitoring
3. Set storage-volume in Prometheus to /apps/prom in deployment.
4. How to install Loki?
5. Open Grafana:
- Set Password
- Set Preference (Organization, time zone, theme)
- Create User
- Add Datasource
- Import Dashboards