CentOS 7環境下Kafka的安裝和基本使用

基礎環境

Windows 10 X64

VMware-workstation-full-12.0.0-2985596

CentOS Linux release 7.4.1708

CentOS 7環境下安裝Kafka

安裝JDK

• 在CentOS上將JDK下載到本地.jdk-8u152-linux-x64.tar.gz
• 解壓並將文件移動到/opt/java目錄下

sudo tar -vxzf jdk-8u152-linux-x64.tar.gz
sudo mv jdk1.8.0_152 /opt/java

• 配置java環境變量

vim /etc/profile
#在profile文件的最後面添加
export JAVA_HOME=/opt/jdk1.8.0_152
export PATH=$JAVA_HOME/bin:$PATH
#添加成功後通過java -version來查看jdk是否安裝成功
java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

安裝Zookeeper

• 通過wget下載zookeeper到本機

sudo wdget https://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

• 解壓並將文件移動到/opt/zookeeper目錄下

# 解壓zookeeper
sudo tar -zxvf zookeeper-3.3.6.tar.gz
# 將zookeeper移動到/opt/zookeeper目錄下
sudo mv zookeeper-3.3.6 /opt/zookeeper

• 編輯zookeeper的配置

# 復制一份zoo_sample.cfg文件並改名為zoo.cfg
sudo cp /opt/zookeeper/zoo_sample.cfg zoo.cfg
 
# 編輯zoo.cfg 文件
sudo vim /opt/zookeeper/zoo.cfg
#主要修改dataDir和server.1=127.0.0.1:2888:3888這2處
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper/data
# the port at which the clients will connect
clientPort=2181
server.1=127.0.0.1:2888:3888

• 配置zookeeper環境變量

sudo vim /etc/profile
#添加如下內容
ZOOKEEPER_HOME=/opt/zookeeper
PATH=$PATH:ZOOKEEPER_HOME/bin
export ZOOKEEPER_HOME

• 啟動zookeeper

zkServer.sh start
# 如果上面命令報錯,則cd /opt/zookeeper/bin 目錄
./zkServer.sh start

安裝kafka

• 通過wget下載kafka到本機

sudo wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.0.0/kafka_2.12-1.0.0.tgz

• 解壓並將文件移動到/opt/kafka目錄下

# 解壓
sudo tar -zxvf kafka_2.12-1.0.0.tgz
# 移動
sudo mv kafka_2.12-1 /opt/kafka

• 編輯kafka的配置

#創建日誌存放目錄
cd /opt/kafka
mkdir -p log/kafka
#修改配置文件/opt/kafka/config/server.properties
sudo vim /opt/kafka/config/server.properties
#主要修改下面幾項內容如下：
broker.id=181
delete.topic.enable=true
listeners = PLAINTEXT://127.0.0.1:9092
log.dirs=/opt/kafka/log/kafka
zookeeper.connect=127.0.0.1:2181

• 配置kafka環境變量

sudo vim /etc/profile
#添加如下內容：
KAFKA_HOME=/opt/kafka
PATH=$PATH:$KAFKA_HOME/bin
export PATH KAFKA_HOME

• 啟動kafka

kafka-server-start.sh /opt/kafka/config/server.properties 

CentOS 7環境下Kafka的基本使用

• 創建一個Topic (test)

kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test

• Producer向Topic發送消息

kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test

[yang@localhost bin]$ ./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>producer send one message
>producer send two message
>         

• Consumer讀取Topic的消息

kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test --from-beginning

[yang@localhost bin]$ ./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 -topic test --from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[2017-11-19 20:32:13,047] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,085] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,100] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
[2017-11-19 20:32:13,265] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
producer send one message
producer send two message

參考資料

CentOS 搭建JDK環境

CentOS7安裝配置ZooKeeper集群

CentOS7.0安裝配置Kafka集群

kafka環境搭建1-單機版

Confluent.Kafka (Kafka .NET client)

CentOS 7環境下Kafka的安裝和基本使用