Kafka單機部署
一般而言，kafka叢集和zookeeper叢集是分開（分機）部署的，我們嘗試一下。
1、參照《Zookeeper叢集環境搭建》搭建zookeeper叢集
環境配置如
$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── cmd
│ ├── start1.sh
│ ├── start2.sh
│ └── start3.sh
└── docker-compose-zookeeper.yml
1 directory, 4 filesdocker-compose-zookeeper.yml
version: '2' networks: thinking: services: test_1.thinking.com: image: ubuntu container_name: test_1.thinking.com command: sh /cmd/start1.sh tty: true networks: - thinking volumes: - /etc/java-8-openjdk:/etc/java-8-openjdk - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64 - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12 - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd environment: - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin test_2.thinking.com: image: ubuntu container_name: test_2.thinking.com command: sh /cmd/start2.sh tty: true networks: - thinking volumes: - /etc/java-8-openjdk:/etc/java-8-openjdk - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64 - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12 - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd environment: - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin test_3.thinking.com: image: ubuntu container_name: test_3.thinking.com command: sh /cmd/start3.sh tty: true networks: - thinking volumes: - /etc/java-8-openjdk:/etc/java-8-openjdk - /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64 - /home/yong/Desktop/zookeeper-3.4.12:/home/zookeeper-3.4.12 - /home/yong/Desktop/kafka-test-20180806/cmd:/cmd environment: - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ - PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin
start1.sh
mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 1 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bashstart2.sh
mkdir -p /tmp/zookeeper
cd /tmp/zookeeper
echo 2 > myid
cd /home/zookeeper-3.4.12/bin/
ln -sf bash /bin/sh
sh zkServer.sh start
/bin/bashstart3.sh
mkdir -p /tmp/zookeeper cd /tmp/zookeeper echo 3 > myid cd /home/zookeeper-3.4.12/bin/ ln -sf bash /bin/sh sh zkServer.sh start /bin/bash
啟動叢集
$ docker-compose -f docker-compose-zookeeper.yml up -d
測試叢集
$ docker exec -it test_1.thinking.com /bin/bash
[email protected]:/# cd /home/zookeeper-3.4.12/bin
[email protected]:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:52:37,981 [myid:] - INFO [main:[email protected] $ docker exec -it test_2.thinking.com /bin/bash
[email protected]:/# cd /home/zookeeper-3.4.12/bin
[email protected]:/home/zookeeper-3.4.12/bin# sh zkCli.sh
Connecting to localhost:2181
2018-08-06 06:55:25,814 [myid:] - INFO [main:[email protected]] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-08-06 06:55:25,817 [myid:] - INFO [main:[email protected]] - Client environment:host.name=53c259066ce4
2018-08-06 06:55:25,817 [myid:] - INFO [main:[email protected]] - Client environment:java.version=1.8.0_171
2018-08-06 06:55:25,819 [myid:] - INFO [main:[email protected]] - Client environment:java.vendor=Oracle Corporation
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:java.class.path=/home/zookeeper-3.4.12/bin/../build/classes:/home/zookeeper-3.4.12/bin/../build/lib/*.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/home/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/home/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/home/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/home/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/home/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/home/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/home/zookeeper-3.4.12/bin/../conf:
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:java.io.tmpdir=/tmp
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:java.compiler=<NA>
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:os.name=Linux
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:os.arch=amd64
2018-08-06 06:55:25,820 [myid:] - INFO [main:[email protected]] - Client environment:os.version=4.13.0-45-generic
2018-08-06 06:55:25,821 [myid:] - INFO [main:[email protected]] - Client environment:user.name=root
2018-08-06 06:55:25,821 [myid:] - INFO [main:[email protected]] - Client environment:user.home=/root
2018-08-06 06:55:25,821 [myid:] - INFO [main:[email protected]] - Client environment:user.dir=/home/zookeeper-3.4.12/bin
2018-08-06 06:55:25,822 [myid:] - INFO [main:[email protected]] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 [email protected]
Welcome to ZooKeeper!
2018-08-06 06:55:25,843 [myid:] - INFO [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-08-06 06:55:25,913 [myid:] - INFO [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-08-06 06:55:25,939 [myid:] - INFO [main-SendThread(localhost:2181):[email protected]] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x200003212ac0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] get /zk-book
123
cZxid = 0x100000002
ctime = Mon Aug 06 06:52:44 GMT 2018
mZxid = 0x100000002
mtime = Mon Aug 06 06:52:44 GMT 2018
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 1] close
2018-08-06 06:55:36,662 [myid:] - INFO [main:[email protected]] - Session: 0x200003212ac0000 closed
[zk: localhost:2181(CLOSED) 2] 2018-08-06 06:55:36,663 [myid:] - INFO [main-EventThread:[email protected]] - EventThread shut down for session: 0x200003212ac0000
[email protected]:/home/zookeeper-3.4.12/bin# 2、啟動kafka伺服器（單機模式）
參考《kafka入門與實踐》（牟大恩）
先編寫docker compose啟動指令碼
$ pwd
/home/yong/Desktop/kafka-test-20180806/
$ tree .
.
├── docker-compose-kafka-sig.yml
└── kafka-sig-cmd
└── kafka-sig.sh
1 directory, 2 files
docker-compose-kafka-sig.yml
version: '2'
networks:
thinking:
services:
kafka.thinking.com:
image: ubuntu
container_name: kafka.thinking.com
command: sh /cmd/kafka-sig.sh
tty: true
networks:
- thinking
volumes:
- /etc/java-8-openjdk:/etc/java-8-openjdk
- /usr/lib/jvm/java-8-openjdk-amd64:/usr/lib/jvm/java-8-openjdk-amd64
- /home/yong/Desktop/kafka-test-20180806/kafka-sig-cmd:/cmd
- /home/yong/Desktop/kafka_2.11-1.1.1:/usr/local/kafka
environment:
- JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
- PATH=${PATH}:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin改寫kafka配置檔案/usr/local/kafka/config/server. properties檔案，主要是改寫
broker.id
host.name
log.dirs
zookeeper.connect
broker.id=1
host.name=kafka.thinking.com
log.dirs=/tmp/kafka-logs
zookeeper.connect=test_1.thinking.com:2181,test_2.thinking.com:2181,test_3.thinking.com:2181
編寫kafka-sig.sh檔案
mkdir -p /tmp/kafka-logs
cd /usr/local/kafka/bin
ln -sf bash /bin/sh
./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
/bin/bash啟動kafka
$ docker-compose -f docker-compose-kafka-sig.yml up -d
檢驗kafka
$ docker exec -it kafka.thinking.com /bin/bash
# cd /usr/local/kafka/bin
# ./kafka-topics.sh --create --zookeeper test_1.thinking.com:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
# ./kafka-topics.sh --zookeeper test_1.thinking.com:2181 --describe --topic test
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
# ./kafka-console-producer.sh --broker-list kafka.thinking.com:9092 --topic test
>Fuck you 1001
>Fuck you 1002
# ./kafka-console-consumer.sh --zookeepertest_1.thinking.com:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
Fuck you 1001
Fuck you 1002
^CProcessed a total of 2 messages成功！
相關推薦
Kafka單機部署
一般而言，kafka叢集和zookeeper叢集是分開（分機）部署的，我們嘗試一下。 1、參照《Zookeeper叢集環境搭建》搭建zookeeper叢集 環境配置如 $ pwd /home/yong/Desktop/kafka-test-20180806/ $ t
centos kafka單包單機部署
centos kafka單包單機部署kafka下載地址http://kafka.apache.org/downloads.html在這裏下載kafka_2.11-0.10.0.0.tgz，kafka和zookeeper都用同一個包裏的安裝和配置jdk(略過)JAVA_HOME=/opt/jdk1.8.0_13
Kafka單機環境的部署
part localhost dev hup ons height factor .tar.gz ip地址 前面說過Kafka集群環境的部署，現在主要說一下在本地測試中Kafka單機環境的部署，和前面一樣首先保證zookeeper服務的正常運行，然後解壓並釋放kafka安裝
kafka單機模式部署安裝，zookeeper啟動
模式 下載 CA jpg str 啟動 ear localhost strong 在root的用戶下 1）：前提 安裝JDK環境，設置JAVA環境變量 2）：下載kafka，命令：wget http://mirrors.shuosc.org/apac
Centos kafka單機配置部署詳解
Centos下kafka 單機配置部署詳解 前提： Linux版本：CentOS release 6.6 (Final) JDK版本：jdk-7u71-linux-x64.tar.gz zookeeper版本：zookeeper-3.3.6.tar.gz
flume+kafka+storm單機部署
flume-1.6.0 kafka0.9.0.0 storm0.9.6 一.部署flume 1.解壓 tar -xzvf apache-flume-1.6.0-bin.tar.gz -C ../app/ 2.在conf目錄中新增自己的配置檔案，內容是source、sink、
linux部署kafka單機叢集環境
一、說明： 作業系統：linux kafka版本資訊：kafka_2.11-0.8.2.1 二、具體操作： 1、安裝kafka之間先檢查作業系統中是否裝有JDK，若沒有點選開啟連結有JDK安裝步驟。 2、關閉SELINUX、開啟防火牆9092
zookeeper與kafka安裝部署及java環境搭建
3.4 項目目錄 tin bytes result zxvf util ise cat 1. ZooKeeper安裝部署 本文在一臺機器上模擬3個zk server的集群安裝。 1.1. 創建目錄、解壓 cd /usr/ #創建項目目錄 mkdir zookeepe
Kafka 環境部署搭建
kafka 環境部署搭建一、基本概念介紹Kafka是一個分布式的、可分區的、可復制的消息系統。它提供了普通消息系統的功能，但具有自己獨特的設計。這個獨特的設計是什麽樣的呢？首先讓我們看幾個基本的消息系統術語：Kafka將消息以topic為單位進行歸納。將向Kafka topic發布消息的程序成為produce
RocketMQ 消息隊列單機部署及使用
post final remove crc infa path dsta ssi slf4 轉載請註明來源：http://blog.csdn.net/loongshawn/article/details/51086876 相關文章： 《Rocke
ETL作業調度軟件TASKCTL4.1單機部署
節點 su命令 權限 一次 hat 作業調度 都是 環境變量 file 單機部署，實際上就是將EM節點和一個Server節點安裝到同一個地方。EM節點是TASKCTL服務端的最頂層，主要負責客戶端與服務端之間的通信。Server節點是TASKCTL的調度服務控制層，也有Ag
Hadoop學習------Hadoop安裝方式之(一)：單機部署
root權限 出錯 密碼登錄 例子 .tar.gz sta /usr 成功 tput Hadoop 默認模式為單機(非分布式模式)，無需進行其他配置即可運行。非分布式即單 Java 進程，方便進行調試。 1、創建用戶 1.1創建hadoop用戶組和用戶 一般我們不會
單機部署Fastfds+nginx
make -m isa entos x64 生成 iptables 創建 可能 一、環境 centos6.5 x64 IP：192.168.66.129 所需軟件包: libfastcommon-1.0.7.zip，FastDFS_v5.05.tar.gz，ngi
kubernetes 1.8 單機部署
kubernetes heapster dashboard docker註:這裏就不科普了，直接開始部署。[這裏使用HTTP來部署]1. 關閉 SeLinux 和 FireWall# sed -i "s/SELINUX=enforcing/SELINUX=disabled" /etc/selinux/co
CentOS 7單機部署Open-Falcon監控
變量 api version rpm包 left 遇到的問題 能夠 pip安裝 vpd 對於Open-Falcon 系統的部署，研究了一段時間才部署成功，文檔主要部署的為v0.2.1版本，參考官網文檔。關於該監控系統的部署分為三個部分，一、環境準備；二、啟動後端；三
centos7單機部署
解壓 pack api .sql txt portal n-n ase 編譯源碼 一、環境準備首先將服務器內部防火墻關閉，並關機開機啟動，並關閉selinux，此處不在贅述。1.1 安裝wget和vim軟件（個人習慣使用vim編輯文件）1yum install -y wge
Kafka單機搭建
rom 1.5 pos listen add eight ati 接收 必須 1、zookeeper搭建 Kafka集群依賴zookeeper，需要提前搭建好zookeeper 單機模式（7步）(集群模式進階請移步：http://blog.51cto.com/n
Minikube之Win10單機部署
本地 ror docker集群 我們 dock mirror 文章 hyper-v管理 tor Kubernetes（k8s）是自動化容器操作的開源平臺，基於這個平臺，你可以進行容器部署，資源調度和集群擴容等操作。如果你曾經用過Docker部署容器，那麽可以將Docker看
【kafka】簡單安裝kafka單機版
kafka topic 1.安裝kafka下載地址 : https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgzrz tar zxf kafka_2.11-0.9.0.0.tgz mv kafka_
MySQL5.6單機部署安裝
erro table align log-error pda init his eve selinux MySQL二進制包安裝： 安裝包獲取方式 官網下載url: http://dev.mysql.com/downloads/mysql/ 數據庫下載url