spark執行例子eclipse maven打包jar
首先在eclipse Java EE中新建一個Maven project具體選項如下
點擊Finish創建成功，接下來把默認的jdk1.5改成jdk1.8
然後編輯pom.xml加入spark-core依賴
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
然後拷貝書中的源碼樣例程序，由於書中spark版本為1.2我的環境spark是2.2.1所以需要修改代碼適應新版本spark API
JavaRDD<String> words = input.flatMap(
new FlatMapFunction<String, String>() {
public Iterator<String> call(String x) {
return Arrays.asList(x.split(" ")).iterator();
}});
然後執行Maven install接下來可以進入目錄E:\developtools\eclipse-jee-neon-3-win32\workspace\learning-spark-mini-example\target找到learning-spark-mini-example-0.0.1-SNAPSHOT.jar並上傳到spark2.2.1環境的linux目錄
然後在linux中執行如下命令，如下圖
[root@hserver1 ~]# spark-submit \
> --class com.oreilly.learningsparkexamples.mini.java.WordCount \
> learning-spark-mini-example-0.0.1-SNAPSHOT.jar \
> /opt/spark-2.2.1-bin-hadoop2.7/README.md wordcounts
spark執行例子eclipse maven打包jar