spark編譯（官方文件翻譯版）
原文地址：http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution
Building Apache Spark
Apache Maven
The Maven-based build is the build of reference for Apache Spark. Building Spark using Maven requires Maven 3.3.9 or newer and Java 8+. Note that support for Java 7 was removed as of Spark 2.2.0.
編譯spark需要maven3.3.9和java7以上。編譯spark2.2.0的話，至少要java8了。
Setting up Maven’s Memory Usage
You’ll need to configure Maven to use more memory than usual by setting MAVEN_OPTS:
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
(The ReservedCodeCacheSize setting is optional but recommended.) If you don’t add these parameters to MAVEN_OPTS
[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.11/classes...
[ERROR] Java heap space -> [Help 1]
You can fix these problems by setting the MAVEN_OPTS variable as discussed before.
Note:
- If using
build/mvnwith noMAVEN_OPTSset, the script will automatically add the above options to theMAVEN_OPTSenvironment variable. - The
testphase of the Spark build will automatically add these options toMAVEN_OPTS, even when not usingbuild/mvn.
build/mvn
Spark now comes packaged with a self-contained Maven installation to ease building and deployment of Spark from source located under the build/ directory. This script will automatically download and setup all necessary build requirements (Maven, Scala, and Zinc) locally within the build/ directory itself. It honors any mvn binary if present already, however, will pull down its own copy of Scala and Zinc regardless to ensure proper version requirements are met. build/mvn execution acts as a pass through to the mvn call allowing easy transition from previous build methods. As an example, one can build a version of Spark as follows:
./build/mvn -DskipTests clean package
Other build examples can be found below.
在spark原始碼的build目錄下，spark現在自帶maven安裝，可以很容易的編譯和部署spark。build/mvn這個指令碼會自動下載和設定必須的編譯環境，maven，scala等。當然如果已經存在了mvn也沒問題，只是可能會不滿足最適當的版本需求。一個最簡單例子如下：
./build/mvn -DskipTests clean package
Building a Runnable Distribution
To create a Spark distribution like those distributed by the Spark Downloads page, and that is laid out so as to be runnable, use ./dev/make-distribution.sh in the project root directory. It can be configured with Maven profile settings and so on like the direct Maven build. Example:
./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn
This will build Spark distribution along with Python pip and R packages. For more information on usage, run ./dev/make-distribution.sh --help
想要編譯一個下載頁面那種分散式的spark，解壓展開後即可執行。就使用原始碼根目錄下的dev/make-distribution.sh指令。它可以配置maven的profile設定，以及其他資訊，就像maven直接編譯一樣。想要獲取更多的這個指令的使用資訊，執行，dev/make-distribution.sh --help。
Specifying the Hadoop Version and Enabling YARN
You can specify the exact version of Hadoop to compile against through the hadoop.version property. If unset, Spark will build against Hadoop 2.6.X by default.
You can enable the yarn profile and optionally set the yarn.version property if it is different from hadoop.version.
Examples:
# Apache Hadoop 2.6.X ./build/mvn -Pyarn -DskipTests clean package # Apache Hadoop 2.7.X and later ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean packag
指定特定的hadoop版本，並開啟spark on yarn。如果不指定的話，就預設匹配hadoop2.6.x的版本，如果yarn的hadoop的版本不一樣，可以用yarn.version屬性來指定（但是我們下載hadoop都是hdfs和yarn是一套的，版本都是一致的）。
Building With Hive and JDBC Support
To enable Hive integration for Spark SQL along with its JDBC server and CLI, add the -Phive and Phive-thriftserver profiles to your existing build options. By default Spark will build with Hive 1.2.1 bindings.
# With Hive 1.2.1 support
./build/mvn -Pyarn -Phive -Phive-thriftserver -DskipTests clean package
想要開啟sparksql的hive整合，和spark的jdbc服務和CLI，就把-Phive 和-Phive-thriftserver兩個配置，加到已有的編譯選項上，預設編譯對hive1.2.1的繫結（如果hive版本更高呢？例如hive2.1.1呢？也沒說怎麼辦，也不知道是不是也可以將就用）。
Packaging without Hadoop Dependencies for YARN
The assembly directory produced by mvn package will, by default, include all of Spark’s dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The hadoop-provided profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
mvn包產生的歸類目錄會包含幾乎所有的spark依賴，包括對Hadoop的依賴和spark生態圈中其他專案的依賴。在on yarn模式下，會導致多個這些版本出現在執行類路徑下：spark assembly和每一個節點上，這些版本的包都被包含在了yarn.application.classpath中。hadoop提供的編譯配置屬性，就不包含hadoop生態圈中的專案，例如zookeeper和hadoop（這部分我也有點沒繞清楚，慚愧！！！）
Building with Mesos support
./build/mvn -Pmesos -DskipTests clean package編譯對mesos的支援
Building for Scala 2.10
To produce a Spark package compiled with Scala 2.10, use the -Dscala-2.10 property:
./dev/change-scala-version.sh 2.10
./build/mvn -Pyarn -Dscala-2.10 -DskipTests clean package
Note that support for Scala 2.10 is deprecated as of Spark 2.1.0 and may be removed in Spark 2.2.0.
編譯對scala2.10的支援，用 -Dscala-2.10配置選項，注意的是，從spark2.1.0開始，對2.10就deprecated（丟棄）了，可能2.2.0之後就不再支援了
Building submodules individually
It’s possible to build Spark sub-modules using the mvn -pl option.
For instance, you can build the Spark Streaming module using:
./build/mvn -pl :spark-streaming_2.11 clean install
where spark-streaming_2.11 is the artifactId as defined in streaming/pom.xml file.
單獨安裝spark子模組，使用-pl配置選項，例如安裝spark-streaming 模組。
Continuous Compilation
We use the scala-maven-plugin which supports incremental and continuous compilation. E.g.
./build/mvn scala:cc
should run continuous compilation (i.e. wait for changes). However, this has not been tested extensively. A couple of gotchas to note:
-
it only scans the paths
src/mainandsrc/test(see docs), so it will only work from within certain submodules that have that structure. -
you’ll typically need to run
mvn installfrom the project root for compilation within specific submodules to work; this is because submodules that depend on other submodules do so via thespark-parentmodule).
Thus, the full flow for running continuous-compilation of the core submodule may look more like:
$ ./build/mvn install
$ cd core
$ ../build/mvn scala:cc使用scala-maven-plugin外掛能夠支援增量的持續編譯，然而沒有被廣泛的測試。有兩點要注意，1）它值掃描src/main和src/test目錄下，所以它將只對有that structure（這個我也不知道什麼）的子模組起作用。2）（沒弄明白- -）。
Building with SBT
Maven is the official build tool recommended for packaging Spark, and is the build of reference. But SBT is supported for day-to-day development since it can provide much faster iterative compilation. More advanced developers may wish to use SBT.
The SBT build is derived from the Maven POM files, and so the same Maven profiles and variables can be set to control the SBT build. For example:
./build/sbt package
To avoid the overhead of launching sbt each time you need to re-compile, you can launch sbt in interactive mode by running build/sbt, and then run all build commands at the command prompt.
maven是官方推薦的編譯工具，但是sbt也被越來越多的人喜愛。sbt也可以用maven的pom檔案，來控制sbt的編譯。為了避免超過負荷launching sbt，你可以通過執行build/sbt來使用互動模式，再在命令列，執行所有的編譯命令。
Speeding up Compilation
Developers who compile Spark frequently may want to speed up compilation; e.g., by using Zinc (for developers who build with Maven) or by avoiding re-compilation of the assembly JAR (for developers who build with SBT). For more information about how to do this, refer to the Useful Developer Tools page.
加速編譯，可以通過zinc（使用maven編譯時）或者避免重複編譯相似的jar（使用sbt編譯時）。更多資訊，參考Useful Developer Tools page.
Encrypted Filesystems
When building on an encrypted filesystem (if your home directory is encrypted, for example), then the Spark build might fail with a “Filename too long” error. As a workaround, add the following in the configuration args of the scala-maven-plugin in the project pom.xml:
<arg>-Xmax-classfile-name</arg>
<arg>128</arg>
and in project/SparkBuild.scala add:
scalacOptions in Compile ++= Seq("-Xmax-classfile-name", "128"),
to the sharedSettings val. See also this PR if you are unsure of where to add these lines.
當在加密的檔案系統裡面編譯spark時，可能會報filename too long的錯誤，可以在pom.xml中給scala-maven-plugin加上如上引數。並且在sparkbuild.scala中機上如上配置。
IntelliJ IDEA or Eclipse
For help in setting up IntelliJ IDEA or Eclipse for Spark development, and troubleshooting, refer to the Useful Developer Tools page.
要整合IDEA或者eclipse參考Useful Developer Tools page.
Running Tests
一下就是一些執行測試的，不再翻譯了。估計很少人會用把。Tests are run by default via the ScalaTest Maven plugin. Note that tests should not be run as root or an admin user.
The following is an example of a command to run the tests:
./build/mvn test
Testing with SBT
The following is an example of a command to run the tests:
./build/sbt test
Running Individual Tests
For information about how to run individual tests, refer to the Useful Developer Tools page.
PySpark pip installable
If you are building Spark for use in a Python environment and you wish to pip install it, you will first need to build the Spark JARs as described above. Then you can construct an sdist package suitable for setup.py and pip installable package.
cd python; python setup.py sdist
Note: Due to packaging requirements you can not directly pip install from the Python directory, rather you must first build the sdist package as described above.
Alternatively, you can also run make-distribution with the –pip option.
PySpark Tests with Maven
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.
./build/mvn -DskipTests clean package -Phive
./python/run-tests
The run-tests script also can be limited to a specific Python version or a specific module
./python/run-tests --python-executables=python --modules=pyspark-sql
Note: You can also run Python tests with an sbt build, provided you build Spark with Hive support.
Running R Tests
To run the SparkR tests you will need to install the knitr, rmarkdown, testthat, e1071 and survival packages first:
R -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival'), repos='http://cran.us.r-project.org')"
You can run just the SparkR tests using the command:
./R/run-tests.sh
Running Docker-based Integration Test Suites
In order to run Docker integration tests, you have to install the docker engine on your box. The instructions for installation can be found at the Docker site. Once installed, the docker service needs to be started, if not already running. On Linux, this can be done by sudo service docker start.
./build/mvn install -DskipTests
./build/mvn test -Pdocker-integration-tests -pl :spark-docker-integration-tests_2.11
or
./build/sbt docker-integration-tests/test