Kafka集群安裝和配置(二)
1、服務器的相關設置:
1、kafka1:172.20.67.52
2、kafka2:172.20.67.56
3、kafka3:172.20.67.57
2、zookeeper所需的三個端口
1、2181
2、2888
3、3888
三個端口的作用:
2181:對客戶端提供服務
2888:集群內機器通訊使用(Leader監聽此端口)
3888:選舉leader使用
3、kafka 所需的端口
9092:kafka所需的端口
(二)、kafka集群的安裝配置
(1)、kafka1172.20.67.52的安裝及相關配置
1、去官網下載相應的版本(https://kafka.apache.org/downloads )。我已下載好直接使用kafka_2.11-0.11.0.3版本
[root@localhost opt]# tar xf kafka_2.11-0.11.0.3.tgz
[root@localhost opt]# mv kafka_2.11-0.11.0.3 /usr/local/kafka
3、zookeeper的相關配置
[root@localhost kafka]# vim /usr/local/kafka/config/zookeeper.properties dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 tickTime=2000 initLimit=20 syncLimit=10 server.1=172.20.67.52:2888:3888 server.2=172.20.67.56:2888:3888 server.3=172.20.67.57:2888:3888
4、kafka的相關配置
[root@localhost kafka]# vim /usr/local/kafka/config/server.properties broker.id=1 host.name=172.20.67.52 listeners=PLAINTEXT://172.20.67.52:9092 log.dirs=/data/kafka-logs log.retention.check.interval.ms=300000 log.retention.hours=1 log.segment.bytes=1073741824 num.io.threads=4 default.replication.factor=2 num.network.threads=3 num.partitions=30 num.recovery.threads.per.data.dir=1 port=9092 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 zookeeper.connect=172.20.67.52:2181,172.20.67.56:2181,172.20.67.57:2181 zookeeper.connection.timeout.ms=10000 fetch.message.max.bytes=52428700 replica.fetch.max.bytes=52428700
5、啟動時加入到/etc/rc.local
[root@localhost config]# vim /etc/rc.local
cd /var/tmp;nohup /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
cd /tmp;nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
touch /var/lock/subsys/local
6、依次創建zookeeper的數據目錄/data/zookeeper和文件/data/zookeeper/myid,此時我們需要對應的Zookeeper服務器上在dataDir的目錄下面新建myid文件,同時把對應的1、2、3寫進去。
[root@localhost config]# mkdir -p /data/zookeeper
[root@localhost config]# vim /data/zookeeper/myid
1
7、依次啟動集群的kafka1、kafka2、kafka3 的zookeeper [root@localhost config]# /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
8、然後在依次啟動集群的kafka1、kafka2、kafka3 的kafka服務[root@localhost config]# cd /tmp;nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
9、最後依次按照2、3、4、5、6、7、8的步驟依次在kafka2(172.20.67.56)和kafka3(172.20.67.57)依次安裝配置zookeeper和kafka的配置文件和啟動。
10、查看集群所啟動的端口
kafka1
[root@Kafka1 ~]# netstat -lntp|grep java
tcp6 0 0 172.20.67.52:3888 :::* LISTEN 20184/java
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 20143/java
tcp6 0 0 172.20.67.52:9092 :::* LISTEN 20649/java
tcp6 0 0 :::35300 :::* LISTEN 20649/java
tcp6 0 0 :::2181 :::* LISTEN 20184/java
tcp6 0 0 :::46471 :::* LISTEN 20184/java
kafka2
[root@Kafka2 ~]# netstat -lntp|grep java
tcp6 0 0 172.20.67.56:3888 :::* LISTEN 30298/java
tcp6 0 0 :::33395 :::* LISTEN 30298/java
tcp6 0 0 :::43031 :::* LISTEN 30573/java
tcp6 0 0 172.20.67.56:9092 :::* LISTEN 30573/java
tcp6 0 0 :::2181 :::* LISTEN 30298/java
kafka3
[root@Kafka3 ~]# netstat -lntp|grep java
tcp6 0 0 172.20.67.57:3888 :::* LISTEN 26313/java
tcp6 0 0 :::44194 :::* LISTEN 26313/java
tcp6 0 0 172.20.67.57:9092 :::* LISTEN 26883/java
tcp6 0 0 :::2181 :::* LISTEN 26313/java
tcp6 0 0 172.20.67.57:2888 :::* LISTEN 26313/java
tcp6 0 0 :::36041 :::* LISTEN 26883/java
Kafka集群安裝和配置(二)