文章目录

• • • • k8s 日志收集方案
      • 1、elasticsearch安装配置
      • • 1.1 es安装
        • 1.2 es配置
        • 1.3 启动es
      • 2、kibana安装配置
      • • 2.1 kibana安装
        • 2.2 kibana配置
        • 2.3 启动kibana
      • 3、zookeeper安装配置
      • • 3.1 zookeeper安装
        • 3.2 启动zookeeper
        • 3.3 检查zookeeper状态
      • 4、kafka安装配置
      • • 4.1 kafka安装
        • 4.2 配置kafka
        • 4.3 启动kafka
      • 5、安装配置logstash
      • • 5.1 安装logstash
        • 5.2 启动logstash
      • 6、配置日志收集
      • • 6.1 基于daemonset的日志收集
        • 6.2 构建logstash镜像
        • 6.3 部署logstash daemonset，yaml如下
        • 6.4 修改logstash配置
        • 6.5 配置kibana展示日志
      • 7、基于sidecar容器的日志收集
      • • 7.1 构建sidecar镜像
        • 7.2 构建镜像
        • 7.3 部署pod
        • 7.4 配置logstash
      • 8、基于容器内置的日志收集进程的日志收集
      • • 8.1 构建镜像
        • 8.2 部署业务容器
        • 8.3 配置logstash
      • 9、elasticsearch-head安装
      • 10、kafka命令

k8s 日志收集方案

三种收集方案的优缺点：

日志收集环境：

k8s日志收集架构

k8s日志收集所用到的安装包及软件：

链接：https://pan.baidu.com/s/1G8XB6dKP8nmgcGNdup_j6g?pwd=vpwu 
提取码：vpwu 

1、elasticsearch安装配置

1.1 es安装

ip:192.168.75.170
rpm -ivh elasticsearch-7.6.2-x86_64.rpm

1.2 es配置

[root@es es]# cat /etc/elasticsearch/elasticsearch.yml| grep -v "#"
cluster.name: log-cluster1
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.75.170
http.port: 9200
discovery.seed_hosts: ["192.168.75.170"]
cluster.initial_master_nodes: ["node-1"]
action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "*"

1.3 启动es

systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch
systemctl status elasticsearch

2、kibana安装配置

2.1 kibana安装

ip：192.168.75.171
rpm -ivh kibana-7.6.2-x86_64.rpm

2.2 kibana配置

[root@kinaba ~]# cat /etc/kibana/kibana.yml | grep -Ev "^#|^$"
server.port: 5601
server.host: "192.168.75.171"
server.name: "kibana"
elasticsearch.hosts: ["http://192.168.75.170:9200"]

2.3 启动kibana

systemctl enable kibana
systemctl start kibana
systemctl status kibana

3、zookeeper安装配置

3.1 zookeeper安装

ip:192.168.75.173

mkdir /usr/local/java
tar xf jdk-8u171-linux-x64.tar.gz -C /usr/local/java/vim /etc/profile	#配置环境变量,添加下面几行
export JAVA_HOME=/usr/local/java/jdk1.8.0_171
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATHsource /etc/profile
java -version

下载安装包，下载地址：https://zookeeper.apache.org/releases.html

  wget https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gzmkdir /appstar xf apache-zookeeper-3.6.3-bin.tar.gz -C /apps/ && cd /appsln -s apache-zookeeper-3.6.3-bin zookeeper

修改zookeeper配置

mkdir -p /data/zookeeper/{data,logs}
cd /apps/zookeeper/
cp conf/zoo_sample.cfg conf/zoo.cfg

cat conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.0=192.168.75.173:2288:3388

echo 0 >/data/zookeeper/data/myid #此处myid在3个节点分别为0,1,2

3.2 启动zookeeper

cat /etc/systemd/system/zookeeper.service
[Unit]
Description=zookeeper.service
After=network.target[Service]
Type=forking
ExecStart=/apps/zookeeper/bin/zkServer.sh start
ExecStop=/apps/zookeeper/bin/zkServer.sh stop
ExecReload=/apps/zookeeper/bin/zkServer.sh restart[Install]
WantedBy=multi-user.target

system daemon reload
systemctl restart zookeeper
systemctl status zookeeper

3.3 检查zookeeper状态

[root@kafka ~]# /apps/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /apps/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

4、kafka安装配置

4.1 kafka安装

和zookeeper部署在相同的主机，下载安装包，下载地址：https://kafka.apache.org/downloads

wget https://archive.apache.org/dist/kafka/3.2.1/kafka_2.12-3.2.1.tgz
tar xf kafka_2.12-3.2.1.tgz -C /apps/ && cd /apps
ln -s kafka_2.12-3.2.1 kafka

4.2 配置kafka

mkdir -p /data/kafka/kafka-logs
cd /app/kafka
vim config/server.properties

broker.id=0	#修改id，3个节点分别为0、1、2
listeners=PLAINTEXT://192.168.75.173:9092	#修改监听地址为本机地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka-logs		#修改数据存放目录
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.75.173:2181	#指定zookeeper地址
zookeeper.connection.timeout.ms=18000`在这里插入代码片`
group.initial.rebalance.delay.ms=0

4.3 启动kafka

cat /etc/systemd/system/kafka.service
[Unit]
Description=kafka.service
After=network.target remote-fs.target zookeeper.service[Service]
Type=forking
Environment=JAVA_HOME=/usr/local/java/jdk1.8.0_171
ExecStart=/apps/kafka/bin/kafka-server-start.sh -daemon /apps/kafka/config/server.properties
ExecStop=/apps/kafka/bin/kafka-server-stop.sh
ExecReload=/bin/kill -s HUP $MAINPID[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start kafka
systemctl status kafka

使用Kafka 客户端管理工具 Offset Explorer查看验证kafka集群状态，关于此工具的使用可以参考：https://blog.csdn.net/qq_39416311/article/details/123316904

5、安装配置logstash

5.1 安装logstash

ip:192.168.75.172
rpm -ivh logstash-7.6.2.rpm

5.2 启动logstash

systemctl start logstash
systemctl enable logstash

6、配置日志收集

6.1 基于daemonset的日志收集

6.2 构建logstash镜像

在k8s或者其他安装有docker的机器上构建logstash镜像，Dockerfile如下

[root@master Dockerfile]# pwd
/root/Dockerfile
[root@master Dockerfile]# cat Dockerfile 
FROM logstash:7.6.2
LABEL author="admin@qq.com"
WORKDIR /usr/share/logstash
COPY logstash.yml /usr/share/logstash/config/logstash.yml
COPY logstash.conf /usr/share/logstash/pipeline/logstash.conf
USER root
RUN usermod -a -G root logstash

logstash.yml内容如下

[root@master Dockerfile]# cat logstash.yml 
http.host: "0.0.0.0"

logstash.conf内容如下

[root@master Dockerfile]# cat logstash.conf 
input {file {path => "/var/lib/docker/containers/*/*-json.log" #docker#path => "/var/log/pods/*/*/*.log"	#使用containerd时，Pod的log的存放路径start_position => "beginning"type => "applog"	#日志类型，自定义}file {path => "/var/log/*.log"	#操作系统日志路径start_position => "beginning"type => "syslog"}
}output {if [type] == "applog" {	#指定将applog类型的日志发送到kafka的哪个topickafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384  #logstash每次向ES传输的数据量大小,单位为字节codec => "${CODEC}"	#日志格式} }if [type] == "syslog" {	##指定将syslog类型的日志发送到kafka的哪个topickafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384codec => "${CODEC}" #系统日志不是json格式}}
}

构建镜像

[root@master Dockerfile]# ls
Dockerfile  logstash.conf  logstash.yml
docker build -t logstash-daemonset:7.6.2 .

打包镜像，传到其他节点，也可以上传到harbor

docker save -o logstash-daemonset.tar logstash-daemonset:7.6.2
docker load -i logstash-daemonset.tar

6.3 部署logstash daemonset，yaml如下

[root@master ~]# cat logstash.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:name: logstash-daemonsetnamespace: log
spec:selector:matchLabels:app: logstashtemplate:metadata:labels:app: logstashspec:tolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedulecontainers:- name: logstashimage: logstash-daemonset-docker:7.6.2imagePullPolicy: IfNotPresentenv:- name: KAFKA_SERVERvalue: "192.168.75.173:9092"- name: TOPIC_IDvalue: "logstash-log-test1"- name: CODECvalue: "json"volumeMounts:- name: varlogmountPath: /var/logreadOnly: False- name: varlogpods# mountPath: /var/log/podsmountPath: /var/lib/docker/containersreadOnly: Falsevolumes:- name: varloghostPath:path: /var/log- name: varlogpodshostPath:#path: /var/log/podspath: /var/lib/docker/containers

[root@master ~]# kubectl apply -f logstash.yaml 
daemonset.apps/logstash-daemonset created
[root@master ~]# kubectl get pods -n log
NAME                       READY   STATUS    RESTARTS   AGE
logstash-daemonset-8j622   1/1     Running   0          7s
logstash-daemonset-dxnwk   1/1     Running   0          7s

在kafka中进行查看，可以看到日志已经发送到kafka

6.4 修改logstash配置

这里的logstash是之前部署在主机上的logstash，而不是Pod。配置其从kafka读取日志然后发送到es

[root@logstash conf.d]# cat logstash-daemonset-kafka-to-es.conf 
input {kafka {bootstrap_servers => "192.168.75.173:9092"topics => ["logstash-log-test1"]codec => "json"}
}output {if [type] == "applog" {elasticsearch {hosts => ["192.168.75.170:9200"]index => "applog-%{+YYYY.MM.dd}"}}if [type] == "syslog" {elasticsearch {hosts => ["192.168.75.170:9200"]index => "syslog-%{+YYYY.MM.dd}"}}
}

检查语法
./logstash -f /etc/logstash/conf.d/logstash-daemonset-kafka-to-es.conf -t
启动
./logstash -f /etc/logstash/conf.d/logstash-daemonset-kafka-to-es.conf
systemctl restart logstash

6.5 配置kibana展示日志

分别为applog和syslog创建日志索引模式





创建pod查看是否有日志上传

[root@master ~]# cat nginx1.yaml 
apiVersion: v1
kind: Pod
metadata:name: nginx2namespace: loglabels:app: nginx2
spec:containers:- name: nginx2image: nginx:latestimagePullPolicy: IfNotPresentports:- containerPort: 80

7、基于sidecar容器的日志收集

在这种方式下node节点上的日志还是需要部署额外的服务去收集

7.1 构建sidecar镜像

[root@master Dockerfile-sidecar]# cat Dockerfile 
FROM logstash:7.6.2
LABEL author="admin@163.com"
WORKDIR /usr/share/logstash
COPY logstash.yml /usr/share/logstash/config/logstash.yml
COPY logstash.conf /usr/share/logstash/pipeline/logstash.conf
USER root
RUN usermod -a -G root logstash

logstash.yaml内容如下

[root@master Dockerfile-sidecar]# cat logstash.yml 
http.host: "0.0.0.0"

logstash.conf内容如下：

[root@master Dockerfile-sidecar]# cat logstash.conf 
input {file {path => "/var/log/applog/catalina.*.log"start_position => "beginning"type => "tomcat-app1-catalina-log"}file {path => "/var/log/applog/localhost_access_log.*.txt"start_position => "beginning"type => "tomcat-app1-access-log"}
}output {if [type] == "tomcat-app1-catalina-log" {kafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384  #logstash每次向ES传输的数据量大小,单位为字节codec => "${CODEC}"} }if [type] == "tomcat-app1-access-log" {kafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384codec => "${CODEC}" #系统日志不是json格式}}
}

7.2 构建镜像

docker build -t logstash-sidecar:7.6.2 .
docker save -o logstash-sidecar.tar logstash-sidecar:7.6.2

7.3 部署pod

[root@master ~]# cat sidecar.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:namespace: logname: tomcat-app1
spec:replicas: 2selector:matchLabels:app: tomcat-app1template:metadata:labels:app: tomcat-app1spec:containers:- name: logstash-sidecarimage: logstash-sidecar:7.6.2imagePullPolicy: IfNotPresentenv:- name: KAFKA_SERVERvalue: "192.168.75.173:9092"- name: TOPIC_IDvalue: "tomcat-log1"- name: CODECvalue: "json"volumeMounts:- name: applogmountPath: /var/log/applog- name: tomcatimage: daocloud.io/library/tomcat:8.0.45imagePullPolicy: IfNotPresentvolumeMounts:- name: applogmountPath: /usr/local/tomcat/logs/volumes:- name: applogemptyDir: {}

[root@master ~]# kubectl get pods -n log
NAME                          READY   STATUS    RESTARTS   AGE
tomcat-app1-55759b549-d6tgv   2/2     Running   0          117s
tomcat-app1-55759b549-r8dfd   2/2     Running   0          117s

在kafka中查看，sidecar容器已经将日志发送到kafka

7.4 配置logstash

cat logstash-sidercat-kafka-to-es.conf
input {kafka {	#从kafka tomcat-app1-log topic中读取日志bootstrap_servers => "192.168.75.173:9092"topics => ["tomcat-app1"]codec => "json"}
}output {if [type] == "tomcat-app1-access-log" {	#tomcat访问日志存储到es的tomcat-app1-accesslog-%{+YYYY.MM.dd}索引中elasticsearch {hosts => ["192.168.75.170:9200"]index => "tomcat-app1-accesslog-%{+YYYY.MM.dd}"}}if [type] == "tomcat-app1-catalina-log" {		#tomcat启动日志存储到es的tomcat-app1-catalinalog-%{+YYYY.MM.dd}索引中elasticsearch {hosts => ["192.168.75.170:9200"]index => "tomcat-app1-catalinalog-%{+YYYY.MM.dd}"}}
}

systemctl restart logstash

8、基于容器内置的日志收集进程的日志收集

8.1 构建镜像

业务镜像内需要运行两个进程，一个是tomcat提供web服务，另一个是filebeat负责收集日志。Dockerfile如下:

[root@node1 Dockerfile]# cat Dockerfile 
FROM centos:centos7.7.1908
WORKDIR /tmp
COPY jdk-8u171-linux-x64.tar.gz /tmp
RUN tar zxf /tmp/jdk-8u171-linux-x64.tar.gz -C /usr/local/ && rm -rf /tmp/jdk-8u171-linux-x64.tar.gz
RUN ln -s /usr/local/jdk1.8.0_171 /usr/local/jdk
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME=${JAVA_HOME}/jre
ENV CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
ENV PATH=${JAVA_HOME}/bin:$PATH
COPY apache-tomcat-8.5.87.tar.gz /tmp
RUN tar zxf apache-tomcat-8.5.87.tar.gz -C /usr/local && rm -rf /tmp/apache-tomcat-8.5.87.tar.gz
RUN mv /usr/local/apache-tomcat-8.5.87 /usr/local/tomcat
COPY filebeat-7.6.2-x86_64.rpm /tmp/
RUN cd /tmp/ && rpm -ivh filebeat-7.6.2-x86_64.rpm && rm -f filebeat-7.6.2-x86_64.rpm
COPY filebeat.yml /etc/filebeat/
COPY run.sh /usr/local/bin/
RUN chmod 755 /usr/local/bin/run.sh
EXPOSE 8443 8080
CMD ["/usr/local/bin/run.sh"]

filebeat.yml内容如下

[root@node1 Dockerfile]# cat filebeat.yml 
filebeat.inputs:
- type: logenabled: truepaths:- /usr/local/tomcat/logs/catalina.*.logfields:type: filebeat-tomcat-catalina
- type: logenabled: truepaths:- /usr/local/tomcat/logs/localhost_access_log.*.txtfields:type: filebeat-tomcat-accesslog
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: false
setup.template.settings:index.number_of_shards: 1
setup.kibana:output.kafka:hosts: ["192.168.75.173:9092"]required_acks: 1topic: "filebeat-tomcat-app1"compression: gzipmax_message_bytes: 1000000

run.sh内容如下：

[root@node1 Dockerfile]# cat run.sh 
#!/bin/bash/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat &/usr/local/tomcat/bin/startup.sh && tail -f /usr/local/tomcat/logs/catalina.out

执行构建

docker build -t filebeat-log:7.6.2 .

8.2 部署业务容器

apiVersion: apps/v1
kind: Deployment
metadata:name: tomcat-myapp
spec:replicas: 3selector:matchLabels:app: tomcat-myapptemplate:metadata:labels:app: tomcat-myappspec:containers:- name: tomcat-myappimage: filebeat-log:7.6.2imagePullPolicy: IfNotPresentports:- name: httpcontainerPort: 8080- name: httpscontainerPort: 8443

[root@master ~]# kubectl get pods -n log
NAME                            READY   STATUS    RESTARTS   AGE
tomcat-myapp-78ff79cd6c-4dnsb   1/1     Running   0          3m4s
tomcat-myapp-78ff79cd6c-mpl4f   1/1     Running   0          3m4s

8.3 配置logstash

cat /etc/logstash/conf.d/filebeat-process-kafka-to-es.confinput {kafka {bootstrap_servers => "192.168.75.173:9092"topics => ["filebeat-tomcat-app1"]codec => "json"}
}output {if [fields][type] == "filebeat-tomcat-catalina" {elasticsearch {hosts => ["192.168.75.170:9200"]index => "filebeat-tomcat-catalina-%{+YYYY.MM.dd}"}}if [fields][type] == "filebeat-tomcat-accesslog" {elasticsearch {hosts => ["192.168.75.170:9200"]index => "filebeat-tomcat-accesslog-%{+YYYY.MM.dd}"}}
}

systemctl restart logstash

9、elasticsearch-head安装

yum install git npm # npm在epel源中
git clone https://github.com/mobz/elasticsearch-head.git # 安装过程需要连接互联网
cd elasticsearch-head # git clone后会自动生成的一个目录
npm install
npm run start

如果想查询集群健康信息，那么需要在elasticsearch配置文件中授权

vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true # elasticsearch中启用CORS
http.cors.allow-origin: "*" # 允许访问的IP地址段，* 为所有IP都可以访问

10、kafka命令

查看已创建的topic列表：
./kafka-topics.sh --list --zookeeper localhost
查看对应topic的描述信息
[root@kafka bin]# ./kafka-topics.sh --describe --zookeeper 192.168.75.173 --topic logstash-log
Topic:logstash-log	PartitionCount:1	ReplicationFactor:1	Configs:Topic: logstash-log	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
消费消息：
[root@kafka bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.75.173:9092 --topic logstash-log --from-beginning[root@kafka bin]# ./kafka-console-producer.sh --broker-list 192.168.75.173:9092 --topic logstash-log
[root@kafka bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.75.173:9092 --topic logstash-log --from-beginning查询topic列表
./kafka-topics.sh --list --zookeeper 192.168.75.173:2181