Flink-on-yarn
阿新 • • 發佈:2018-11-23
解壓 stat master swd run rec 地址 abi man
介紹
官網下載
- https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop28-scala_2.11.tgz
flink集群地址
- ddcve: master
- dddve: slave
- ddeve: slave
解壓
tar -xzvf flink-1.6.1-bin-hadoop28-scala_2.11.tgz
mv flink-1.6.1 flink
cd flink
修改配置
- 指定master
- vim conf/flink-conf.yaml
jobmanager.rpc.address: ddcve # 指定master
- 指定slaves
echo -e "dddve\nddeve" > conf/slaves
添加yarn配置項
- hadoop/etc/hadoop/yarn-site.xml
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>
啟動flink
./bin/start-cluster.sh
- 輸出結果如下
Starting HA cluster with 2 masters. Starting standalonesession daemon on host ddcvc. Starting standalonesession daemon on host ddcve. Starting taskexecutor daemon on host dddve. Starting taskexecutor daemon on host ddeve.
flink on yarn
以下操作都在slave
- 啟動yarn-session
./bin/yarn-session.sh -d -n 4 -tm 2048 -s 4
- 提交job
# 先在hdfs創建輸入文件
hadoop fs -copyFromLocal /tmp/passwd /tmp/passwd
# 提交
./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/passwd
flink with zookeeper
- 修改conf/flink-conf.yaml
high-availability: zookeeper high-availability.storageDir: hdfs:///flink/ha/ high-availability.zookeeper.quorum: ddcva:2181,dddva:2181,ddeva:2181 recovery.zookeeper.path.root: /flink high-availability.zookeeper.client.acl: open state.backend: filesystem
- 修改conf/masters
ddcvc:8081
ddcve:8081
Flink-on-yarn