CentOS下部署kakfa叢集

zookeeper版本：zookeeper-3.4.13

kakfa版本：kafka_2.11-2.0.0

1 環境準備

在已有zookeeper叢集上，部署kakfa叢集。

zookeeper叢集主機資訊如下：

2 安裝kafka

2.1 下載kafka

下載地址：http://mirrors.hust.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

wget http://mirrors.hust.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

解壓kafka

tar -zxvf kafka_2.11-2.0.0.tgz

移動到/root/apps下並重命名kafka

mv kafka_2.11-2.0.0 /root/apps/kafka

2.2 修改配置檔案

以下操作只在master主機上進行，完成後將kafka目錄scp到其它兩臺機子

2.2.1 建立日誌目錄

進入kafka目錄

cd /root/apps/kafka

在此目錄建立kafkalogs目錄用來存放kafka的日誌資料

mkdir kafkalogs

2.2.2 修改配置檔案

vi /config/server.properties

修改配置說明

broker.id=0 # 設定broker.id為0,其它兩個節點需修改為1，2
listeners=PLAINTEXT://:9092
 
 #監聽埠為9092
advertised.listeners=PLAINTEXT://master.hadoop.com:9092 # 設定hostname
log.dirs=/root/apps/kafka/kafkalogs # 日誌資料儲存位置
zookeeper.connect=master.hadoop.com:2181,slave1.hadoop.com:2181,slave2.hadoop.com:2181 # zookeeper的ip和埠

2.3 主機分發

在master主機上將修改後的kafka資料夾分發到其它兩臺主機上,並修改配置檔案中的broker.id 和hostname

scp -r /root/apps/kafka/ slave1.hadoop.com:/root/apps/
scp -r /root/apps/kafka/ slave2.hadoop.com:/root/apps/

2.4 啟動kafka

啟動kakfa之前，先啟動zookeeper。然後再啟動kafka

sh /root/apps/kafka/bin/kafka-server-start.sh -daemon /root/apps/kafka/config/server.properties

啟動指令碼

#!/bin/bash
#master
echo 'start master zookeeper'
sh /root/apps/zookeeper/bin/zkServer.sh start
echo 'start slave1 zookeeper'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/zookeeper/bin/zkServer.sh start
exit
eeooff
echo 'start slave2 zookeeper'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/zookeeper/bin/zkServer.sh start
exit
eeooff
echo 'start mastet kafka'
sh /root/apps/kafka/bin/kafka-server-start.sh -daemon /root/apps/kafka/config/server.properties
echo 'start slave1 kafka'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/kafka/bin/kafka-server-start.sh -daemon /root/apps/kafka/config/server.properties
exit
eeooff
echo 'start slave2 kafka'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/kafka/bin/kafka-server-start.sh -daemon /root/apps/kafka/config/server.properties
exit
eeooff

停止指令碼

#!/bin/bash
# stop kafka
echo 'stop master kafka'
sh /root/apps/kafka/bin/kafka-server-stop.sh
echo 'stop slave1 kafka'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/kafka/bin/kafka-server-stop.sh
exit
eeooff
echo 'stop slave2 kafka'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/kafka/bin/kafka-server-stop.sh
exit
eeooff
# stop zookeeper
echo 'stop master zookeeper'
sh /root/apps/zookeeper/bin/zkServer.sh stop
echo 'stop slave1 zookeeper'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/zookeeper/bin/zkServer.sh stop
exit
eeooff
echo 'stop slave2 zookeeper'
ssh [email protected] > /dev/null 2>&1 << eeooff
sh /root/apps/zookeeper/bin/zkServer.sh stop
exit
eeooff