大資料 Flink 1.8 最新版本使用
阿新 • • 發佈：2019-01-10
Flink 1.8 的快照版本已經發布，我們可以來使用一下，當然了，過程肯定還是有一些曲折的，這裡已經幫大家給記錄下來了
使用過程
下載
git clone https://github.com/apache/flink
編譯（大概有20分鐘這樣）
cd flink
mvn clean package -DskipTests
編譯通過版本在build-target目錄中
[[email protected] flink]$ ls build-target
bin conf examples lib LICENSE log NOTICE opt README.txt
直接執行cluster
[[email protected] bin]$ ./start-cluster.sh Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Starting cluster. Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Starting standalonesession daemon on host storm1.starsriver.cn. Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set. Starting taskexecutor daemon on host storm1.starsriver.cn.
執行在yarn上,還需要做一些變動
lib中需要如下幾個包 打包下載
flink-hadoop-compatibility_2.11-1.8-SNAPSHOT.jar
javax.ws.rs-api-2.0.1.jar
jersey-common-2.27.jar
jersey-core-1.9.jar
就可以啟動yarn-session.sh了
bin/yarn-session.sh run -n 2 -tm 2048 -s 4
或者是單個yarn-job
flink run -d -m yarn-cluster ./dataset-1.0-SNAPSHOT-all.jar -yn 2 -ytm 2048m