1. 程式人生 > >使用kafka和zookeeper 構建分散式編譯環境

使用kafka和zookeeper 構建分散式編譯環境

1:在每臺機器上安裝jdk, 指令碼程式碼如下:

每一個機器上下載jdk,zookeeper,kafka

連結:https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

      http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.13/

     http://mirrors.shu.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

2:部署java的環境,設定環境變數

sudo tar -zxvf jdk-8u191-linux-x64.tar.gz -C /opt/

echo "export JAVAHOME=\"/opt/jdk1.8.0_191\"
export PATH=\"\$PATH:\$JAVAHOME/bin\"
export CLASSPATH=\".:\$JAVAHOME/lib:\$JAVAHOME/jre/lib\"
" >> ~/.bashrc

source ~/.bashrc

java -version

 3:部署zookeeper環境

sudo tar -zxvf zookeeper-3.4.13.tar.gz -C /opt/

#設定環境變數
export PATH="$PATH:$JAVAHOME/bin:/opt/zookeeper-3.4.13/bin:/opt/kafka_2.11-2.0.0/bin"

#三臺伺服器的環境配置
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/var/log/zookeeper

server.1=109.123.100.126:2888:3888
server.2=109.123.100.137:2888:3888
server.3=109.123.100.139:2888:3888

 

#新建目錄並設定許可權
sudo mkdir -p /var/log/zookeeper && sudo mkdir -p /data/zookeeper

sudo chmod -R a+w /var/log/zookeeper && sudo chmod -R a+w /data/zookeeper

 

#server 126
echo "1" > /data/zookeeper/myid

#server 137
echo "2" > /data/zookeeper/myid

#server 139
echo "3" > /data/zookeeper/myid

 

4:部署kafka環境

sudo tar -xvf kafka_2.11-2.0.0.tgz -C /opt

#設定環境變數
export PATH="$PATH:$JAVAHOME/bin:/opt/zookeeper-3.4.13/bin:/opt/kafka_2.11-2.0.0/bin"

 

#啟動kafka叢集,在每個機器上執行如下命令
kafka-server-start.sh -daemon /opt/kafka_2.11-2.0.0/config/server.properties

 

#通過jps 驗證結果
[email protected]:~$ jps
5090 Jps
3650 QuorumPeerMain
5007 Kafka
#看到三個程序,說明成功了

 5:建立topic 模擬多個生產者一個消費者,和多個消費者,一個生產者

建立一個tizen-unified的topic, 表示這個topic中放的都是tizen-unified 的資訊

kafka-topics.sh --create --replication-factor 2 --zookeeper 109.123.100.126:2181,109.123.100.137:2181,109.123.100.144:2181 --partitions 1 --topic tizen-unified

 開啟一個生產者:

kafka-console-producer.sh --broker-list 109.123.100.126:9092 --topic tizen-unified

[email protected]:~$ kafka-console-producer.sh --broker-list 109.123.100.126:9092 --topic tizen-unified
>buildint package1
>building package2
>building package3
>building package4
>building package5

 開啟兩個消費者:(同屬於一個group)

kafka-console-consumer.sh --bootstrap-server 109.123.100.126:9092 --topic tizen-unified --from-beginning --group tizen-worker

[email protected]:~$ kafka-console-consumer.sh --bootstrap-server 109.123.100.126:9092 --topic tizen-unified --from-beginning --group tizen-worker
building package3
building package4
building package5

[email protected]:~$ kafka-console-consumer.sh --bootstrap-server 109.123.100.126:9092 --topic tizen-unified --from-beginning --group tizen-worker
building package1
building package2