1、安裝scala：

在SparkNode01~03節點安裝：

cd /opt

tar xzvf scala-2.11.12.tgz

mv scala-2.11.12 scala

scp -r scala @SparkNode03:/opt

vi /etc/profile

SCALA_HOME=/opt/scala

PATH=$PATH:$SCALA_HOME/bin

source /etc/profile

scala -version

2、安裝Spark

在SparkNode01~03節點安裝：

cd /opt

tar xzvf spark-2.2.1-bin-hadoop2.7.tgz

mv spark-2.2.1-bin-hadoop2.7 spark

cd spark/conf

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

添加：

export JAVA_HOME=/usr/local/java/jdk1.8.0_121

export SCALA_HOME=/opt/scala

export HADOOP_HOME=/opt/hadoop

export HADOOP_CONF_DIR=/opt/hadoop/tc/hadoop

export SPARK_MASTER_PORT=7077

export SPARK_EXECUTOR_CORES=2

export SPARK_EXECUTOR_MEMORY=1024m

export SPARK_EXECUTOR_INSTANCES=1

cp slaves.template slaves

vi slaves

添加：

SparkNode01

SparkNode02

SparkNode03

在hdfs上傳spark的jar包，並修改/opt/spark/conf/spark-defaults.conf

cd /opt/hadoop/bin

./hadoop fs -mkdir /spark_jars

./hadoop fs -put /opt/spark/jars/* /spark_jars

cp spark-defaults.conf.template spark-defaults.conf

vi spark-defaults.conf

添加：

spark.yarn.jars=hdfs://Master01:9000/spark_jars/*

驗證：

./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --num-executors 3 /opt/spark/examples/jars/spark-examples_2.11-2.2.1.jar 10

SPARK 2.2.1 YARN安裝部署