作者：周志湖
微訊號：zhouzhihubeyond

本節為下一節Kafka與Spark Streaming做鋪墊

主要內容

1.kafka 叢集搭建

1. kafka 叢集搭建

1. kafka 安裝與配置

tar -zxvf  kafka_2.10-0.8.2.1.tgz 

解壓，解壓後的目錄如下

進入config目錄，將server.properties檔案內容如下：

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
 

broker.id=0

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkmaster

//中間省略，預設配置即可
############################# Zookeeper ###
 
##########################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
 

zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

將整個安裝檔案進行跨機器拷貝：

root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave01:/hadoopLearning/ 
root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave02:/hadoopLearning/ 

將sparkslave01機器上的server.properties檔案內容如下：

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave01

//中間省略，預設配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

將sparkslave02機器上的server.properties檔案內容如下：

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave02


//中間省略，預設配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

1. 啟動Kafka叢集

root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 
root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 
root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 

3 建立topic
在sparkmaster機器上執行下列命令建立一個topic

root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-topics.sh --create --topic kafkatopictest --replication-factor 3 --partitions 2 --zookeeper sparkmaster:2181
Created topic "kafkatopictest".

4 傳送訊息至kafka
在sparkslave01機器上執行下列命令並向kafka傳送訊息

root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-producer.sh --broker-list sparkslave01:9092 --sync --topic kafkatopictest
Hello Kafka, I will test Spark Streaming on you next lesson

5 接收kafka傳送來的訊息

在sparkslave02機器上執行下列命令並接收kafka傳送訊息

root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-consumer.sh --zookeeper sparkmaster:2181 --topic kafkatopictest --from-beginning
Hello Kafka, I will test Spark Streaming on you next lesson

至此Kafka 叢集搭建與測試完畢

下一節當中，我們將演示kafka如何與Spark Streaimg結合起來使用