使用Kube-Prometheus部署k8s监控

概述

在k8s中部署Prometheus监控的方法

通常在k8s中部署prometheus监控可以采取的方法有以下三种:

  1. 通过yaml手动部署
  2. operator部署
  3. 通过helm chart部署

kube-prometheus和Prometheus Operator的关系

prometheus-operator官方地址:https://github.com/prometheus-operator/prometheus-operator
kube-prometheus官方地址:https://github.com/prometheus-operator/kube-prometheus

两个项目的关系:前者只包含了Prometheus Operator,后者既包含了Prometheus Operator,又包含了Prometheus相关组件的部署及常用的Prometheus自定义监控,具体包含下面的组件

The Prometheus Operator:创建CRD自定义的资源对象
Highly available Prometheus:创建高可用的Prometheus
Highly available Alertmanager:创建高可用的告警组件
Prometheus node-exporter:创建主机的监控组件
Prometheus Adapter for Kubernetes Metrics APIs:创建自定义监控的指标工具(例如可以通过nginx的request来进行应用的自动伸缩)
kube-state-metrics:监控k8s相关资源对象的状态指标
Grafana:进行图像展示

为什么用Prometheus Operator

Prometheus Operator的本职就是一组用户自定义的CRD资源以及Controller的实现,Prometheus Operator负责监听这些自定义资源的变化,并且根据这些资源的定义自动化的完成如Prometheus Server自身以及配置的自动化管理工作。以下是Prometheus Operator的架构图

准备k8s环境

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[[email protected] ~]# kubectl version -o yaml
clientVersion:
buildDate: "2020-11-11T13:17:17Z"
compiler: gc
gitCommit: d360454c9bcd1634cf4cc52d1867af5491dc9c5f
gitTreeState: clean
gitVersion: v1.19.4
goVersion: go1.15.2
major: "1"
minor: "19"
platform: linux/amd64
serverVersion:
buildDate: "2020-11-11T13:09:17Z"
compiler: gc
gitCommit: d360454c9bcd1634cf4cc52d1867af5491dc9c5f
gitTreeState: clean
gitVersion: v1.19.4
goVersion: go1.15.2
major: "1"
minor: "19"
platform: linux/amd64
[[email protected] ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 83d v1.19.4
k8s-node01 Ready <none> 83d v1.19.4
k8s-node02 Ready <none> 83d v1.19.4
k8s-node03 Ready <none> 83d v1.19.4
k8s-node04 Ready <none> 83d v1.19.4
[[email protected] ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
[[email protected] ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 23d
1
2
3
4
5
6
# git clone https://github.com/prometheus-operator/kube-prometheus.git
[[email protected] ~]# cd kube-prometheus/
[[email protected] kube-prometheus]# git checkout release-0.7
[[email protected] kube-prometheus]# git branch
main
* release-0.7

添加CRD

部署完成后,会创建一个名为monitoring的 namespace,所以资源对象对将部署在改命名空间下面

1
2
3
4
5
6
7
8
9
10
11
12
[[email protected] manifests]# pwd
/root/kube-prometheus/manifests
[[email protected] manifests]# kubectl apply -f setup/
[[email protected] manifests]# kubectl get crd |grep coreos
alertmanagerconfigs.monitoring.coreos.com 2021-03-02T05:45:05Z
alertmanagers.monitoring.coreos.com 2021-03-02T05:45:05Z
podmonitors.monitoring.coreos.com 2021-03-02T05:45:05Z
probes.monitoring.coreos.com 2021-03-02T05:45:05Z
prometheuses.monitoring.coreos.com 2021-03-02T05:45:05Z
prometheusrules.monitoring.coreos.com 2021-03-02T05:45:06Z
servicemonitors.monitoring.coreos.com 2021-03-02T05:45:06Z
thanosrulers.monitoring.coreos.com 2021-03-02T05:45:06Z

部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[[email protected] manifests]# pwd
/root/kube-prometheus/manifests
[[email protected] manifests]# kubectl apply -f .
[[email protected] manifests]# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 76s
alertmanager-main-1 2/2 Running 0 76s
alertmanager-main-2 2/2 Running 0 76s
grafana-f8cd57fcf-l6d7g 1/1 Running 0 74s
kube-state-metrics-587bfd4f97-kqhss 3/3 Running 0 74s
node-exporter-6z2g2 2/2 Running 0 75s
node-exporter-ckwh6 2/2 Running 0 75s
node-exporter-dl56x 2/2 Running 0 75s
node-exporter-ftt58 2/2 Running 0 75s
node-exporter-gq626 2/2 Running 0 75s
prometheus-adapter-69b8496df6-rrs56 1/1 Running 0 75s
prometheus-k8s-0 2/2 Running 1 75s
prometheus-k8s-1 2/2 Running 1 75s
prometheus-operator-7649c7454f-fwgd2 2/2 Running 0 88s
[[email protected] manifests]# kubectl get svc -n monitoring # 检查SVC
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main ClusterIP 10.96.184.62 <none> 9093/TCP 107s
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 107s
grafana NodePort 10.98.138.229 <none> 3000:30100/TCP 106s
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 106s
node-exporter ClusterIP None <none> 9100/TCP 106s
prometheus-adapter ClusterIP 10.107.27.129 <none> 443/TCP 106s
prometheus-k8s NodePort 10.99.142.87 <none> 9090:30200/TCP 106s
prometheus-operated ClusterIP None <none> 9090/TCP 106s
prometheus-operator ClusterIP None <none> 8443/TCP 2m

解决ControllerManager、Scheduler监控问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[[email protected] manifests]# cat prometheus-kube-controller-manager-service.yaml 
apiVersion: v1
kind: Service
metadata:
name: kube-controller-manager
namespace: kube-system
labels:
k8s-app: kube-controller-manager
spec:
selector:
component: kube-controller-manager
ports:
- name: https-metrics
port: 10257
targetPort: 10257
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: kube-controller-manager
namespace: kube-system
labels:
k8s-app: kube-controller-manager
subsets:
- addresses:
- ip: 10.62.169.145 # 此ip为k8s-master的ip地址
ports:
- name: https-metrics
port: 10257
protocol: TCP

[[email protected] manifests]# kubectl apply -f prometheus-kube-controller-manager-service.yaml

修改kube-controller-manager监听地址

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[[email protected]master manifests]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
...
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=0.0.0.0
...
[[email protected]master manifests]# netstat -lntup|grep kube-contro
tcp6 0 0 :::10252 :::* LISTEN 21186/kube-controll
tcp6 0 0 :::10257 :::* LISTEN 21186/kube-controll

同理给kube-scheduler创建对应的SVC和Endpoints,修改监听地址

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[[email protected] manifests]# cat prometheus-kube-scheduler-service.yaml 
apiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-scheduler
labels:
k8s-app: kube-scheduler
spec:
selector:
component: kube-scheduler
ports:
- name: http-metrics
port: 10259
targetPort: 10259
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: kube-scheduler
namespace: kube-system
labels:
k8s-app: kube-scheduler
subsets:
- addresses:
- ip: 10.62.169.145
ports:
- name: https-metrics
port: 10259
protocol: TCP

[[email protected] manifests]# kubectl apply -f prometheus-kube-scheduler-service.yaml

修改kube-controller-manager监听地址

1
2
3
4
5
6
7
8
9
10
11
12
13
[[email protected] manifests]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
...
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=0.0.0.0
...
[[email protected] manifests]# netstat -lntup|grep kube-sche
tcp6 0 0 :::10251 :::* LISTEN 20672/kube-schedule
tcp6 0 0 :::10259 :::* LISTEN 20672/kube-schedule

关闭Watchdog报警

Watchdog报警是一个正常的报警,这个告警的作用是:如果alermanger或者prometheus本身挂掉了就发不出告警了,因此一般会采用另一个监控来监控prometheus,或者自定义一个持续不断的告警通知,哪一天这个告警通知不发了,说明监控出现问题了。prometheus operator已经考虑了这一点,本身携带一个watchdog,作为对自身的监控
如果需要关闭,删除或注释掉Watchdog部分

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[[email protected] manifests]# vim prometheus-rules.yaml   # 注释以下内存
...
- name: general.rules
rules:
- alert: TargetDown
annotations:
message: '{{ printf "%.4g" $value }}% of the {{ $labels.job }}/{{ $labels.service }} targets in {{ $labels.namespace }} namespace are down.'
expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job, namespace, service)) > 10
for: 10m
labels:
severity: warning
# - alert: Watchdog
# annotations:
# message: |
# This is an alert meant to ensure that the entire alerting pipeline is functional.
# This alert is always firing, therefore it should always be firing in Alertmanager
# and always fire against a receiver. There are integrations with various notification
# mechanisms that send a notification when this alert is not firing. For example the
# "DeadMansSnitch" integration in PagerDuty.
# expr: vector(1)
# labels:
# severity: none
...
[[email protected] manifests]# kubectl apply -f prometheus-rules.yaml

再次登录prometheus界面

PrometheusOperator为什么要数据持久化

增加prometheus数据持久化,是因为在日常中如果重启Prometheus Pod后,会发现之前的数据不见了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[[email protected] manifests]# kubectl get pod prometheus-k8s-0 -n monitoring -o yaml
...
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
...
volumes:
...
- emptyDir: {}
name: prometheus-k8s-db
...

可以看到 Prometheus 的数据目录 /prometheus 实际上是通过 emptyDir 进行挂载的,我们知道 emptyDir 挂载的数据的生命周期和 Pod 生命周期一致的,所以如果 Pod 挂掉了,数据也就丢失了,这也就是为什么我们重建 Pod 后之前的数据就没有了的原因

这里我使用nfs后端创建StorageClass对象,创建的nfs-client-provisioner方式请看 https://www.lijiawang.org/posts/k8s%E4%BD%BF%E7%94%A8nfs%E5%81%9Aprovisioner/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[[email protected] manifests]# kubectl get storageclasses # 提前准备storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 23d
[[email protected] manifests]# vim prometheus-prometheus.yaml
...
spec:
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.22.1
retention: 3d
storage:
volumeClaimTemplate:
spec:
storageClassName: managed-nfs-storage
resources:
requests:
storage: 5Gi
[[email protected] manifests]# kubectl apply -f prometheus-prometheus.yaml

查看pvc,pv,pod的状态

1
2
3
4
5
6
7
8
[[email protected] manifests]# kubectl get pvc,pv -n monitoring  |grep prometheus
persistentvolumeclaim/prometheus-k8s-db-prometheus-k8s-0 Bound pvc-c6f133ea-c524-48a8-91a2-f7534446f831 5Gi RWO managed-nfs-storage 43m
persistentvolumeclaim/prometheus-k8s-db-prometheus-k8s-1 Bound pvc-f4ce8966-6529-4d85-9562-ea4ad8d62e6e 5Gi RWO managed-nfs-storage 43m
persistentvolume/pvc-c6f133ea-c524-48a8-91a2-f7534446f831 5Gi RWO Delete Bound monitoring/prometheus-k8s-db-prometheus-k8s-0 managed-nfs-storage 43m
persistentvolume/pvc-f4ce8966-6529-4d85-9562-ea4ad8d62e6e 5Gi RWO Delete Bound monitoring/prometheus-k8s-db-prometheus-k8s-1 managed-nfs-storage 43m
[[email protected] manifests]# kubectl get pod -n monitoring |grep prometheus-k8s
prometheus-k8s-0 2/2 Running 1 45m
prometheus-k8s-1 2/2 Running 1 45m

验证Prometheus Pod 的数据目录关联到一个 PVC 对象上了

现在我们再去看 Prometheus Pod 的数据目录就可以看到是关联到一个 PVC 对象上了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[[email protected] manifests]# kubectl get pod prometheus-k8s-0 -n monitoring -o yaml
...
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /etc/prometheus/certs
name: tls-assets
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
...
volumes:
...
- name: prometheus-k8s-db
persistentVolumeClaim:
claimName: prometheus-k8s-db-prometheus-k8s-0
...

即使我们的 Pod 挂掉了,数据也不会丢失了!


本博客所有文章除特别声明外,均采用 CC BY-SA 4.0 协议 ,转载请注明出处!