根據生產環境要求，需要採集K8S Pod 日誌，和開發協商之後，Pod中應用會將日誌輸出到容器終端上，這時可以直接用filebeat 採集node節點上面的`/var/log/containers/*.log`日誌，然後將日誌輸出到kafka訊息佇列中，經過kafka將日誌寫入logstash進行格式化，然後由logstash傳入elasticsearch儲存，然後kibana會連線elasticsearch展示索引資料。 資料傳輸流程：**Pod -> `/var/log/containers/*.log` -> Filebeat -> Kafka叢集 -> Logstash -> Elasticsearch -> Kibana** ## K8S 配置Filebeat 整體配置檔案如下： ```shell $ ls filebeat.daemonset.yml filebeat.permission.yml filebeat.indice-lifecycle.configmap.yml filebeat.settings.configmap.yml ``` ### Filebeat操作許可權 ```shell $ cat filebeat.permission.yml --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: filebeat labels: app: filebeat rules: - apiGroups: [""] resources: - namespaces - pods verbs: - get - watch - list --- apiVersion: v1 kind: ServiceAccount metadata: namespace: kube-system name: filebeat labels: app: filebeat ``` ### Filebeat主配置檔案 > 注意：如果收集Java堆疊錯誤日誌，需要增加下面帶註釋的幾行引數，multiline多行處理解決次問題。 ``` $ cat filebeat.settings.configmap.yml --- apiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: filebeat-config labels: app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: container enabled: true paths: - /var/log/containers/*.log multiline: # 多行處理，正則表示如果前面幾個數字不是4個數字開頭，那麼就會合併到一行,解決Java堆疊錯誤日誌收集問題 pattern: ^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2} #匹配Java日誌開頭時間 negate: true # 正則是否開啟，預設false不開啟 match: after # 不匹配的正則的行是放在上面一行的前面還是後面 processors: - add_kubernetes_metadata: in_cluster: true host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" - add_cloud_metadata: - add_kubernetes_metadata: matchers: - logs_path: logs_path: "/var/log/containers/" - add_docker_metadata: output: kafka: enabled: true # 增加kafka的輸出 hosts: ["10.0.0.72:9092"] topic: filebeat max_message_bytes: 5242880 partition.round_robin: reachable_only: true keep-alive: 120 required_acks: 1 setup.ilm: policy_file: /etc/indice-lifecycle.json ``` ### Filebeat索引生命週期策略配置 > ElasticSearch 的 indice 生命週期表示一組規則，可以根據 indice 的大小或者時長應用到你的 indice 上。比如可以每天或者每次超過 1GB 大小的時候對 indice 進行輪轉，我們也可以根據規則配置不同的階段。由於監控會產生大量的資料，很有可能一天就超過幾十G的資料，所以為了防止大量的資料儲存，我們可以利用 indice 的生命週期來配置資料保留，這個在 Prometheus 中也有類似的操作。 如下所示的檔案中，我們配置成每天或每次超過5GB的時候就對 indice 進行輪轉，並刪除所有超過30天的 indice 檔案，我們這裡只保留30天監控資料完全足夠了。 ``` filebeat.indice-lifecycle.configmap.yml --- apiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: filebeat-indice-lifecycle labels: app: filebeat data: indice-lifecycle.json: |- { "policy": { "phases": { "hot": { "actions": { "rollover": { "max_size": "5GB" , "max_age": "1d" } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } } ``` ### Filebeat Daemonset配置檔案 ``` $ cat filebeat.daemonset.yml --- apiVersion: apps/v1 kind: DaemonSet metadata: namespace: kube-system name: filebeat labels: app: filebeat spec: selector: matchLabels: app: filebeat template: metadata: labels: app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 containers: - name: filebeat image: docker.elastic.co/beats/filebeat:7.8.0 args: [ "-c", "/etc/filebeat.yml", "-e", ] env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName securityContext: runAsUser: 0 resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: filebeat-indice-lifecycle mountPath: /etc/indice-lifecycle.json readOnly: true subPath: indice-lifecycle.json - name: data mountPath: /usr/share/filebeat/data - name: varlog mountPath: /var/log readOnly: true - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: dockersock mountPath: /var/run/docker.sock volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: filebeat-indice-lifecycle configMap: defaultMode: 0600 name: filebeat-indice-lifecycle - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: dockersock hostPath: path: /var/run/docker.sock - name: data hostPath: path: /var/lib/filebeat-data type: DirectoryOrCreate ``` ### 執行到K8S中 ``` $ kubectl apply -f filebeat.settings.configmap.yml \ -f filebeat.indice-lifecycle.configmap.yml \ -f filebeat.daemonset.yml \ -f filebeat.permissions.yml configmap/filebeat-config created configmap/filebeat-indice-lifecycle created daemonset.apps/filebeat created clusterrolebinding.rbac.authorization.k8s.io/filebeat created clusterrole.rbac.authorization.k8s.io/filebeat created serviceaccount/filebeat cre