找個spark叢集搭建是針對於上一篇hadoop的基礎上搭建的。
所以spark的版本也是要按照著hadoop版本進行下載。

1.解壓spark，修改spark的/etc/profile的home目錄。

2.安裝SCALA，並配置SCALA_HOME。

3.修改spark conf目錄下的spark-env.sh檔案，並新增下列配置    
    export JAVA_HOME=/root/java/jdk1.8.0_181
    export HADOOP_HOME=/root/hadoop/hadoop-2.7.6
    export HADOOP_CONF_DIR=/root/hadoop/hadoop-2.7.6/etc/hadoop
    export SCALA_HOME=/root/scala/scala-2.11.8
    export SPARK_MASTER_IP=192.168.124.132

4.修改spark conf目錄下的slavers檔案配置：
    centos01
    centos02
    centos03

5.分發spark到其它節點，包括/etc/profile檔案
    scp -r /etc/profile [email protected]:/etc/
    scp -r ~/spark [email protected]:/root/

5.測試spark yarn提交模式，在spark example目錄下有個PI運算jar包
    spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster spark-examples_2.11-2.3.0.jar 10

    利用spark-shell --master yarn進行測試，會出現異常：
    2018-08-27 01:01:30 ERROR SparkContext:91 - Error initializing SparkContext.
    org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
    利用http://192.168.124.132:8088/cluster/app/application_1535337995441_0003：
    檢視Diagnostics（診斷）發現報錯虛擬記憶體超過了：
    is running beyond virtual memory limits.
    Current usage: 40.9 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

    配置yarn-site.xml:
    <!--以下為解決spark-shell 以yarn client模式執行報錯問題而增加的配置,估計spark-summit也會有這個問題。2個配置只用配置一個即可解決問題，當然都配置也沒問題-->
    <!--虛擬記憶體設定是否生效,若實際虛擬記憶體大於設定值 ,spark 以client模式執行可能會報錯,"Yarn application has already ended! It might have been killed or unable to l"-->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
        <description>Whether virtual memory limits will be enforced for containers</description>
    </property>
    <!--配置虛擬記憶體/實體記憶體的值，預設為2.1,實體記憶體預設應該是1g，所以虛擬記憶體是2.1g-->
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4</value>
        <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
    </property>

6.配置spark sql的metastore到mysql進行管理
    檢視機器是否安裝過mysql：rpm -qa|grep -i mysql
    wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
    rpm -ivh mysql57-community-release-el7-8.noarch.rpm
    yum install mysql-community-server

    安裝完成後，重啟mysql：service mysqld restart
    然後檢視初始密碼：grep "password" /var/log/mysqld.log
    登陸mysql，修改密碼~！：alter user 'root'@'localhost' identified by '[email protected]';
    重新整理許可權：flush privileges;

7.把jdbc的jar包複製到spark jars目錄下：（注意：使用5版本的可以避免出現時區問題）
    網上下載mysql jdbc驅動，需要注意驅動和mysql的版本~！
    網上下載https://dev.mysql.com/downloads/connector/j/
    rpm包解壓：rpm2cpio mysql-connector-java-8.0.12-1.el7.noarch.rpm | cpio -div
    把jdbc驅動複製到spark jars目錄：cp mysql-connector-java-8.0.12.jar ~/sparkapp/spark-2.3.0-bin-hadoop2.7/jars/

8.在spark conf目錄下建立一個hive-site.xml檔案：
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hiveMetastore?createDatabaseIfNotExist=true&amp;characterEncoding=utf8&amp;useSSL=false</value>
    <description>hiveMetastore:Metastore sive in mysql</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>e
    <description>mysql account</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>[email protected]</value>
    <description>mysql password</description>
  </property>
</configuration>

9.在mysql上建立相關的spark元資料資料庫，和檢視：
    mysql:
    create database sparkmetastore;
    use sparkmetastore;

    create view databases_v 
    as select DBS.* 
    from hiveMetastore.DBS;

    create view tables_v 
    as select TBLS.*, DBS.NAME 
    from hiveMetastore.DBS, hiveMetastore.TBLS 
    where TBLS.DB_ID=DBS.DB_ID AND TBLS.TBL_TYPE!='VIRTUAL_VIEW';

    create view views_v 
    as select TBLS.*, DBS.NAME 
    from hiveMetastore.DBS, hiveMetastore.TBLS 
    where TBLS.DB_ID=DBS.DB_ID AND TBLS.TBL_TYPE='VIRTUAL_VIEW';

    create view columns_v 
    as select COLUMNS_V2.*, TBLS.TBL_NAME, DBS.NAME 
    from hiveMetastore.DBS, hiveMetastore.TBLS, hiveMetastore.COLUMNS_V2
    where DBS.DB_ID = TBLS.DB_ID AND COLUMNS_V2.CD_ID = TBLS.TBL_ID;

10.再在mysql上建立一個使用者，該使用者為spark-sql jdbc元資料查詢使用者：
    CREATE USER 'spark'@'%' IDENTIFIED BY '[email protected]';
    GRANT SELECT ON sparkmetastore.* TO 'spark'@'%';

11.在spark-sql中進行反向查詢元資料：
    create database spark;
    CREATE TABLE databases_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.databases_v","user" "spark", "password" "[email protected]")
    CREATE TABLE tables_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.tables_v","user" "spark", "password" "[email protected]")
    CREATE TABLE views_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.views_v","user" "spark", "password" "[email protected]")
    CREATE TABLE columns_v USING org.apache.spark.sql.jdbc OPTIONS("url" "jdbc:mysql://192.168.124.132:3306", "dbtable" "sparkmetastore.columns_v","user" "spark", "password" "[email protected]")

12.開啟spark sbin./start-thriftserver.sh服務。

13.在遠端root訪問hdfs的時候，會出現許可權問題：
    在hdfs-site.xml加入找個配置
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>btd-dev-2027700.lvs02.dev.ebayc3.com:50090</value>
    </property>
    <!-- hdfs用root登陸的時候會出現許可權錯誤>
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
        <description>
        If "true", enable permission checking in HDFS.
        If "false", permission checking is turned off,
        but all other behavior is unchanged.
        Switching from one parameter value to the other does not change the mode,
        owner or group of files or directories.
    </description>

14.配置spark-default.conf:(一定要把spark-warehouse設定儲存到hdfs上，不然會出現錯誤)
    spark.master yarn

   spark.sql.warehouse.dir hdfs://centos01:9000/spark-warehouse