前言

首次安裝的環境搭配是這樣的：
jdk8
hadoop2.6.5
spark2.1
scala2.12.4
Anaconda3-5.1.0
一連串的報錯讓人驚喜無限，儘管反覆調整配置始終無法解決。

坑了一整天后最後最終發現是版本不相容！！再次提醒自己一定要重視各元件版本的問題。這裡最主要的是spark和Anaconda版本的相容問題，為了相容python3儘量用新版的spark。最終解決方案的版本搭配如下：
jdk8
hadoop2.7.5
spark2.3.0
scala2.11.12
Anaconda3-5.1.0

一、VM安裝Ubuntu16.04虛擬機器

sudo apt-get update
sudo apt-get install vim
sudo apt-get install openssh-server

# 配置ssh免密登陸
ssh localhost
ssh-keygen -t rsa //一路回車
cat id_rsa.pub >> authorized_keys

sudo vi /etc/hosts //新增各個節點ip
192.168.221.132 master
192.168.221.133 slave1
192.168.221.134 slave2

# sudo vi /etc/hostname
master
 

二、配置profile環境變數

#Java
export JAVA_HOME=/home/hadoop/jdk1.8.0_161
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jar
#Hadoop
export HADOOP_HOME=/home/hadoop/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
#Scala
export SCALA_HOME=/home/hadoop/scala
export PATH=$PATH:$SCALA_HOME/bin
#Anaconda
export PATH=/home/hadoop/anaconda3/bin:$PATH
 

export PYSPARK_DRIVER_PYTHON=/home/hadoop/anaconda3/bin/jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
export PYSPARK_PYTHON=/home/hadoop/anaconda3/bin/python
#Spark
export SPARK_HOME=/home/hadoop/spark
export PATH=$PATH:$SPARK_HOME/bin

三、hadoop 六個配置檔案

# hadoop-env.sh
export JAVA_HOME=/home/hadoop/hadoop/jdk1.8.0_161

# core-site.xml
<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/home/hadoop/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
</configuration>

# hdfs-site.xml
<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>master:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/home/hadoop/hadoop/tmp/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/home/hadoop/hadoop/tmp/dfs/data</value>
        </property>
</configuration>

# mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>master:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master:19888</value>
        </property>
</configuration>

# yarn-site.xml
<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>master</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

# slaves
slave1
slave2

三、spark兩個配置檔案

# spark-env.sh
#java
export JAVA_HOME=/home/hadoop/jdk1.8.0_161
#scala
export SCALA_HOME=/home/hadoop/scala
#hadoop
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
export YARN_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
#spark
export SPARK_HOME=/home/hadoop/spark
export SPARK_LOCAL_DIRS=/home/hadoop/spark
export SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop/bin/hadoop classpath)
export SPARK_WORKER_CORES=1
export SPARK_WORKER_INSTANCES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_MASTER_IP=master
export SPARK_LIBRARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native

# slaves
slave1
slave2

四、解壓縮檔案

scp jdk-8u161-linux-x64.tar hadoop@master:~
scp Anaconda3-5.1.0-Linux-x86_64.sh hadoop@master:~
scp -r hadoop/ hadoop@master:~
scp -r scala/ hadoop@master:~
scp -r spark/ hadoop@master:~

tar -xvf jdk-8u161-linux-x64.tar -C ./

source ~/.profile
分別檢視jdk版本、hadoop版本、scala版本

# 叢集模式啟動spark檢視jps
spark-shell --master spark://master:7077 --executor-memory 512m --total-executor-cores 2

五、安裝Anaconda

bash Anaconda3-5.1.0-Linux-x86_64.sh -b


# 建立配置jupyter_notebook_config.py
jupyter notebook --generate-config
vim ~/.jupyter/jupyter_notebook_config.py

c = get_config()
c.IPKernelApp.pylab = 'inline'
c.NotebookApp.ip = '*' 
c.NotebookApp.open.browser = False
c.NotebookApp.password = u''
c.NotebookApp.port = 8888

六、關機後克隆出兩個新節點並配置相關內容

sudo vi /etc/hostname

sudo vi /etc/hosts

七、遠端測試pyspark叢集

# 伺服器端啟動叢集
start-all.sh
spark/sbin/start-all.sh

# hadoop和spark的程序都顯示正常後開始啟動pyspark
1、local模式執行
pyspark

2、Stand Alone執行模式
MASTER=spark://master:7077 pyspark --num-executors 1 --total-executor-cores 3 --executor-memory 512m

然後在遠端Web端輸入192.168.221.132:8888
頁面開啟後需要輸入驗證資訊（第一次驗證即可）：
輸入上圖token後面的字串和使用者密碼

輸入sc測試

至此，aconda3-5.1.0（Python3.6.4） 搭建pyspark遠端伺服器部署成功。