Spark Streaming的maven配置pom.xml
阿新 • • 發佈:2018-11-02
不多說,直接查詢相關的內容
https://mvnrepository.com/search?q=spark
我們可以看到很多常見的東西,spark-core,spark-sql,spark-streaming等等
點開這些連結進入到版本列表介面,選擇自己需要的配置
spark-core
spark-streaming
這裡就不一一說明了,下面我直接列出一個已經測試成功的pom.xml
<properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <encoding>UTF-8</encoding> <scala.version>2.11.12</scala.version> <spark.version>2.3.0</spark.version> <hadoop.version>2.6.5</hadoop.version> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-common</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-core</artifactId> <version>${hadoop.version}</version> <scope>provided</scope> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>${spark.version}</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>${spark.version}</version> <scope>provided</scope> </dependency> </dependencies>
這裡不建議在dependency中寫死version,而是採用一下方式
<hadoop.version>2.6.5</hadoop.version>
在我的配置中有4個hadoop.version,如果寫死了,那麼後邊要進行統一的hadoop升級,你該如何操作?一個一個找,然後修改4個位置?如果這裡有上百個依賴,你該如何操作。
這個pom.xml修改完畢,儲存後,我的eclipse上爆出了170個problems,但是不要慌,進行如下操作