zookeeper和kafka安裝文件
一：kafka簡介：
Kafka 被稱為下一代分散式訊息系統，是非營利性組織ASF(Apache Software Foundation，簡稱為ASF)基金會中的一個開源專案，比如HTTP Server、Hadoop、ActiveMQ、Tomcat等開源軟體都屬於Apache基金會的開源軟體，類似的訊息系統還有RbbitMQ、ActiveMQ、ZeroMQ，最主要的優勢是其具備分散式功能、並且結合zookeeper可以實現動態擴容。
安裝環境：
三臺伺服器：
IP分別是：192.168.56.11 192.168.56.12 192.168.56.13
三臺伺服器分別配置hosts檔案：
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.11 linux-host1.exmaple.com
192.168.56.12 linux-host2.exmaple.com
192.168.56.13 linux-host3.exmaple.com
1.1：下載安裝並驗證zookeeper：
1.1.1：kafka下載地址：
1.1.2：zookeeper 下載地址：
1.1.3：安裝zookeeper：
zookeeper叢集特性：整個叢集種只要有超過叢集數量一半的zookeeper工作只正常的，那麼整個叢集對外就是可用的，假如有2臺伺服器做了一個zookeeper叢集，只要有任何一臺故障或宕機，那麼這個zookeeper叢集就不可用了，因為剩下的一臺沒有超過叢集一半的數量，但是假如有三臺zookeeper組成一個叢集，那麼損壞一臺就還剩兩臺，大於3臺的一半，所以損壞一臺還是可以正常執行的，但是再損壞一臺就只剩一臺叢集就不可用了。那麼要是4臺組成一個zookeeper叢集，損壞一臺叢集肯定是正常的，那麼損壞兩臺就還剩兩臺，那麼2臺不大於叢集數量的一半，所以3臺的zookeeper叢集和4臺的zookeeper叢集損壞兩臺的結果都是叢集不可用，一次類推5臺和6臺以及7臺和8臺都是同理，所以這也就是為什麼叢集一般都是奇數的原因。
下載後的安裝檔案上傳到各伺服器的/usr/local/src目錄然後分別執行以下操作。
1.1.3.1：Serer1 配置：
[[email protected] ~]# cd /usr/local/src/
[[email protected] src]# yum install jdk-8u151-linux-x64.tar.gz -y
[[email protected] src]# tar xvf zookeeper-3.4.11.tar.gz
[[email protected] src]# ln -sv /usr/local/src/zookeeper-3.4.11 /usr/local/zookeeper
[[email protected] src]# mkdir /usr/local/zookeeper/data
[[email protected] src]# cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# vim /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# grep "^[a-Z]" /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000 #伺服器之間或客戶端與伺服器之間的單次心跳檢測時間間隔，單位為毫秒
initLimit=10 #叢集中leader伺服器與follower伺服器第一次連線最多次數
syncLimit=5 # leader 與 follower 之間傳送和應答時間，如果該follower 在設定的時間內不能與leader 進行通訊，那麼此 follower 將被視為不可用。
clientPort=2181 #客戶端連線 Zookeeper 伺服器的埠，Zookeeper 會監聽這個埠，接受客戶端的訪問請求
dataDir=/usr/local/zookeeper/data #自定義的zookeeper儲存資料的目錄
server.1=192.168.56.11:2888:3888 #伺服器編號=伺服器IP:LF資料同步埠:LF選舉埠
server.2=192.168.56.12:2888:3888
server.3=192.168.56.13:2888:3888
[[email protected] src]# echo "1" > /usr/local/zookeeper/data/myid
1.1.3.2：Server2 配置：
[[email protected] ~]# cd /usr/local/src/
[[email protected] src]# yum install jdk-8u151-linux-x64.tar.gz -y
[[email protected] src]# tar xvf zookeeper-3.4.11.tar.gz
[[email protected] src]# ln -sv /usr/local/src/zookeeper-3.4.11 /usr/local/zookeeper
[[email protected] src]# mkdir /usr/local/zookeeper/data
[[email protected] src]# cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# vim /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# grep "^[a-Z]" /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
clientPort=2181
dataDir=/usr/local/zookeeper/data
server.1=192.168.56.11:2888:3888
server.2=192.168.56.12:2888:3888
server.3=192.168.56.13:2888:3888
[[email protected] src]# echo "2" > /usr/local/zookeeper/data/myid
1.1.3.3：Server3 配置：
[[email protected] ~]# cd /usr/local/src/
[[email protected] src]# yum install jdk-8u151-linux-x64.tar.gz -y
[[email protected] src]# tar xvf zookeeper-3.4.11.tar.gz
[[email protected] src]# ln -sv /usr/local/src/zookeeper-3.4.11 /usr/local/zookeeper
[[email protected] src]# mkdir /usr/local/zookeeper/data
[[email protected] src]# cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# vim /usr/local/zookeeper/conf/zoo.cfg
[[email protected] src]# grep "^[a-Z]" /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
clientPort=2181
dataDir=/usr/local/zookeeper/data
server.1=192.168.56.11:2888:3888
server.2=192.168.56.12:2888:3888
server.3=192.168.56.13:2888:3888
[[email protected] src]# echo "3" > /usr/local/zookeeper/data/myid
1.1.3.4：各伺服器啟動zookeeper：
[[email protected] ~]# /usr/local/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[[email protected] src]# /usr/local/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[[email protected] src]# /usr/local/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
1.1.3.5：檢視各zookeeper狀態：
[[email protected] ~]# /usr/local/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[[email protected] ~]# /usr/local/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[[email protected] ~]# /usr/local/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
1.1.3.6：zookeeper簡單操作命令：
#連線到任意節點生成資料：
[[email protected] data]# /usr/local/zookeeper/bin/zkCli.sh -server 192.168.56.11:2181
[zk: 192.168.56.11:2181(CONNECTED) 3] create /test "hello"
#在其他zookeeper節點驗證資料：
[[email protected] src]# /usr/local/zookeeper/bin/zkCli.sh -server 192.168.56.12:2181
[zk: 192.168.56.12:2181(CONNECTED) 0] get /test
hello
cZxid = 0x100000004
ctime = Fri Dec 15 11:14:07 CST 2017
mZxid = 0x100000004
mtime = Fri Dec 15 11:14:07 CST 2017
pZxid = 0x100000004
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
1.2：安裝並測試kafka：
1.2.1：Server1安裝kafka：
[[email protected] src]# tar xvf kafka_2.11-1.0.0.tgz
[[email protected] src]# ln -sv /usr/local/src/kafka_2.11-1.0.0 /usr/local/kafka
[[email protected] src]# vim /usr/local/kafka/config/server.properties
21 broker.id=1
31 listeners=PLAINTEXT://192.168.56.11:9092
103 log.retention.hours=24 #保留指定小時的日誌內容
123 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 #所有的zookeeper地址
1.2.2：Server2安裝kafka：
[[email protected] src]# tar xvf kafka_2.11-1.0.0.tgz
[[email protected] src]# ln -sv /usr/local/src/kafka_2.11-1.0.0 /usr/local/kafka
[[email protected] src]# vim /usr/local/kafka/config/server.properties
21 broker.id=2
31 listeners=PLAINTEXT://192.168.56.12:9092
123 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181
1.2.3：Server3安裝kafka：
[[email protected] src]# tar xvf kafka_2.11-1.0.0.tgz
[[email protected] src]# ln -sv /usr/local/src/kafka_2.11-1.0.0 /usr/local/kafka
[[email protected] src]# vim /usr/local/kafka/config/server.properties
21 broker.id=3
31 listeners=PLAINTEXT://192.168.56.13:9092
123 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181
1.2.4：分別啟動kafka：
1.2.4.1：Serevr1啟動kafka：
[[email protected] src]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties #以守護程序的方式啟動
1.2.4.2：Serevr2啟動kafka：
[[email protected] src]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
1.2.4.3：Serevr3啟動kafka：
[[email protected] src]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
#/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties & #此方式zookeeper會在shell斷開後關閉
1.2.5：測試kafka:
1.2.5.1：驗證程序：
[[email protected] ~]# jps
10578 QuorumPeerMain
11572 Jps
11369 Kafka
[[email protected] ~]# jps
2752 QuorumPeerMain
8229 Kafka
8383 Jps
[[email protected] ~]# jps
12626 Kafka
2661 QuorumPeerMain
12750 Jps
1.2.5.2：測試建立topic：
建立名為logstashtest，partitions(分割槽)為3，replication(複製)為3的topic(主題)：
在任意kafaka伺服器操作：
[[email protected] ~]# /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --partitions 3 --replication-facto
r 3 --topic logstashtest
Created topic "logstashtest".
1.2.5.3：測試獲取topic：
可以在任意一臺kafka伺服器進行測試：
[[email protected] ~]# /usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --topic logstashtest
狀態說明：logstashtest有三個分割槽分別為1、2、3，分割槽0的leader是3（broker.id），分割槽0有三個副本，並且狀態都為lsr（ln-sync，表示可以參加選舉成為leader）。
1.2.5.4：刪除topic：
[[email protected] ~]# /usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --topic logstashtest
Topic logstashtest is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
1.2.5.5：獲取所有topic：
[[email protected] ~]# /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181
__consumer_offsets
nginx-accesslog-5612
system-log-5612
1.2.6：kafka命令測試訊息傳送：
1.2.6.1：建立topic:
[[email protected] ~]# /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --partitions 3 --replication-factor 3 --topic messagetest
Created topic "messagetest".
1.2.6.2：傳送訊息：
[[email protected] ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.56.11:9092,192.168.56.12:9092,192.168.56.13:9092 --topic messagetest
>hello
>kafka
>logstash
>ss
>oo
1.2.6.3：其他kafka伺服器測試獲取資料：
#Server1：
[[email protected] ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --topic messagetest --from-beginning
#Server2：
#Server3：
1.2.7：使用logstash測試向kafka寫入資料：
1.2.7.1：編輯logstash配置檔案：
[[email protected] ~]# vim /etc/logstash/conf.d/logstash-to-kafka.sh
input {
stdin {}
}
output {
kafka {
topic_id => "hello"
bootstrap_servers => "192.168.56.11:9092"
batch_size => 5
}
stdout {
codec => rubydebug
}
}
1.2.7.2：驗證kafka收到logstash資料：
[[email protected] ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 --topic hello --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] inste
ad of [zookeeper].
2017-12-15T14:33:00.684Z linux-host3.exmaple.com hello
2017-12-15T14:33:31.127Z linux-host3.exmaple.com test
[[email protected] ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.56.11:9
092,192.168.56.12:9092,192.168.56.13:9092 --topic messagetest>hello
>kafka
>logstash