K8S之监控etcd集群(云原生应用自带metrics接口)

K8S之监控etcd集群(云原生应用自带metrics接口)

K8S之监控etcd集群(云原生应用自带metrics接口)

监控etcd集群原理模型

image.png

1. 查看接口信息

$ curl --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem  https://192.168.0.107:2379/metrics -k
# 这样也行
$ curl -L http://localhost:2379/metrics

2. 创建service和Endpoints

# 创建ep和svc代理外部的etcd服务,其他自带metrics接口的服务也是如此!
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: etcd-monitor
  name: etcd-monitor
  namespace: kube-system
subsets:
- addresses:
  - ip: 192.168.0.107
  - ip: 192.168.0.108
  - ip: 192.168.0.109
  ports:
  - name: etcd-metrics
    port: 2379
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: etcd-monitor
  name: etcd-monitor
  namespace: kube-system
spec:
  ports:
  - name: etcd-metrics
    port: 2379
    protocol: TCP
    targetPort: 2379
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

3.测试是否代理成功

#再次curl,把IP换成svc的IP测试,输出相同内容即创建成功
$ kubectl get ep,svc -n kube-system etcd-monitor
NAME                     ENDPOINTS            AGE
endpoints/etcd-monitor   192.168.0.107:2379   16m

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/etcd-monitor   ClusterIP   10.96.219.144   <none>        2379/TCP   13m

# 再次请求接口
$ curl --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem  https://10.96.219.144:2379/metrics -k

4. 创建secret

# 1、这里我们k8s-master01节点进行创建,ca为etcd ca证书,剩下2个为etcd证书和key,这是我证书所在位置
  cert-file: '/etc/etcd/ssl/etcd.pem'
  key-file: '/etc/etcd/ssl/etcd-key.pem'
  ca-file: '/etc/etcd/ssl/etcd-ca.pem'
  
# 2、接下来我们需要创建一个secret,让prometheus pod节点挂载
$ kubectl create secret generic etcd-certs --from-file=/etc/etcd/ssl/etcd-ca.pem --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem -n monitoring


# 3、创建完成后可以检查一下
$ kubectl get secret -n monitoring |grep etcd
etcd-certs                        Opaque                                3      47s
$ kubectl describe secret -n monitoring etcd-certs 
Name:         etcd-certs
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
etcd-ca.pem:   1367 bytes
etcd-key.pem:  1679 bytes
etcd.pem:      1493 bytes

5. 编辑prometheus,把证书挂载进去

# 1、通过edit直接编辑prometheus
$ vim prometheus-prometheus.yaml
# 在replicas底下加上secret名称
replicas:2
secrets:
- etcd-certs  #添加secret名称

$ kubectl replace -f prometheus-prometheus.yaml

# 进入容器查看,就可以看到证书挂载进去了
$ kubectl exec -it -n monitoring prometheus-k8s-0 -- sh

# 查看文件是否存在
/prometheus $ ls /etc/prometheus/secrets/etcd-ssl/
etcd-ca.pem   etcd-key.pem  etcd.pem

6. 创建ServiceMonitor

$ cat etcd-serviceMonitor.yaml 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: etcd-monitor
  name: etcd-monitor
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: etcd-metrics  # 这个port对应 Service.spec.ports.name
    scheme: https
    tlsConfig:
      #证书路径 (在prometheus pod里路径)
      caFile: /etc/prometheus/secrets/etcd-certs/etcd-ca.pem
      certFile: /etc/prometheus/secrets/etcd-certs/etcd.pem
      keyFile: /etc/prometheus/secrets/etcd-certs/etcd-key.pem
      insecureSkipVerify: true   # 关闭客户端证书校验
  selector:
    matchLabels:
      k8s-app: etcd-monitor # 跟scv的lables保持一致
  namespaceSelector:
    matchNames:
    - kube-system  # 跟svc所在namespace保持一致

# 匹配Kube-system这个命名空间下面具有k8s-app=etcd-monitor这个label标签的Serve,job label用于检索job任务名称的标签。由于证书serverName和etcd中签发的证书可能不匹配,所以添加了insecureSkipVerify=true将不再对服务端的证书进行校验

7.页面查看三个etcd节点都获取到数据

image.png

此处数据获取有点慢,需要等待一下
可以在Prometheus的web界面上查看监控的节点信息:Status --> Service Discovery --> serviceMonitor/monitoring/etcd-monitor

image.png

8. grafana模板导入

数据采集完成后,接下来可以在grafana中导入dashboard
# 打开官网来的如下图所示,点击下载JSO文件
grafana官网:https://grafana.com/grafana/dashboards/3070
中文版ETCD集群插件:https://grafana.com/grafana/dashboards/9733
点击HOME–>导入模板

image.png

导入后页面即可展示etcd的监控数据

image.png