Scala進階之路-Spark獨立模式集群部署

阿新 • • 發佈：2018-07-27

cas pro 多人 tro splay park def 你在原來

　　　　　　　　　　Scala進階之路-Spark獨立模式集群部署

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正傑

　　我們知道Hadoop解決了大數據的存儲和計算，存儲使用HDFS分布式文件系統存儲，而計算采用MapReduce框架進行計算，當你在學習MapReduce的操作時，尤其是Hive的時候（因為Hive底層其實仍然調用的MapReduce）是不是覺得MapReduce運行的特別慢？因此目前很多人都轉型學習Spark，今天我們就一起學習部署Spark集群吧。

一.準備環境

　　如果你的服務器還麽沒有部署Hadoop集群，可以參考我之前寫的關於部署Hadoop高可用的筆記：https://www.cnblogs.com/yinzhengjie/p/9154265.html

1>.啟動HDFS分布式文件系統

[yinzhengjie@s101 download]$ more `which xzk.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]

#判斷用戶是否傳參
if [ $# -ne 1 ];then
    echo "無效參數，用法為: $0  {start|stop|restart|status}"
    exit
fi

#獲取用戶輸入的命令
cmd=$1

#定義函數功能
function zookeeperManger(){
     
case $cmd in
    start)
        echo "啟動服務"        
        remoteExecution start
        ;;
    stop)
        echo "停止服務"
        remoteExecution stop
        ;;
    restart)
        echo "重啟服務"
        remoteExecution restart
        ;;
    status)
        echo "查看狀態"
        remoteExecution status
        ;;
     
*)
        echo "無效參數，用法為: $0  {start|stop|restart|status}"
        ;;
    esac
}


#定義執行的命令
function remoteExecution(){
    for (( i=102 ; i<=104 ; i++ )) ; do
            tput setaf 2
            echo ========== s$i zkServer.sh  $1 ================
            tput setaf 9
            ssh s$i  "source /etc/profile ; zkServer.sh $1"
    done
}

#調用函數
zookeeperManger
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ more `which xzk.sh` 技術分享圖片

[yinzhengjie@s101 download]$ more `which xcall.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]


#判斷用戶是否傳參
if [ $# -lt 1 ];then
        echo "請輸入參數"
        exit
fi

#獲取用戶輸入的命令
cmd=$@

for (( i=101;i<=105;i++ ))
do
        #使終端變綠色 
        tput setaf 2
        echo ============= s$i $cmd ============
        #使終端變回原來的顏色，即白灰色
        tput setaf 7
        #遠程執行命令
        ssh s$i $cmd
        #判斷命令是否執行成功
        if [ $? == 0 ];then
                echo "命令執行成功"
        fi
done
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ more `which xcall.sh` 技術分享圖片

[yinzhengjie@s101 download]$ more `which xrsync.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:[email protected]

#判斷用戶是否傳參
if [ $# -lt 1 ];then
    echo "請輸入參數";
    exit
fi


#獲取文件路徑
file=$@

#獲取子路徑
filename=`basename $file`

#獲取父路徑
dirpath=`dirname $file`

#獲取完整路徑
cd $dirpath
fullpath=`pwd -P`

#同步文件到DataNode
for (( i=102;i<=105;i++ ))
do
    #使終端變綠色 
    tput setaf 2
    echo =========== s$i %file ===========
    #使終端變回原來的顏色，即白灰色
    tput setaf 7
    #遠程執行命令
    rsync -lr $filename `whoami`@s$i:$fullpath
    #判斷命令是否執行成功
    if [ $? == 0 ];then
        echo "命令執行成功"
    fi
done
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ more `which xrsync.sh` 技術分享圖片

[yinzhengjie@s101 download]$ xzk.sh start
啟動服務
========== s102 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== s103 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== s104 zkServer.sh start ================
ZooKeeper JMX enabled by default
Using config: /soft/zk/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ xcall.sh jps
============= s101 jps ============
2603 Jps
命令執行成功
============= s102 jps ============
2316 Jps
2287 QuorumPeerMain
命令執行成功
============= s103 jps ============
2284 QuorumPeerMain
2319 Jps
命令執行成功
============= s104 jps ============
2305 Jps
2276 QuorumPeerMain
命令執行成功
============= s105 jps ============
2201 Jps
命令執行成功
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ xzk.sh start

[yinzhengjie@s101 download]$ start-dfs.sh 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Starting namenodes on [s101 s105]
s101: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s101.out
s105: starting namenode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-namenode-s105.out
s103: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s103.out
s104: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s104.out
s102: starting datanode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-datanode-s102.out
Starting journal nodes [s102 s103 s104]
s104: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s104.out
s102: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s102.out
s103: starting journalnode, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-journalnode-s103.out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Starting ZK Failover Controllers on NN hosts [s101 s105]
s101: starting zkfc, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-zkfc-s101.out
s105: starting zkfc, logging to /soft/hadoop-2.7.3/logs/hadoop-yinzhengjie-zkfc-s105.out
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ xcall.sh jps
============= s101 jps ============
7909 Jps
7526 NameNode
7830 DFSZKFailoverController
命令執行成功
============= s102 jps ============
2817 QuorumPeerMain
4340 JournalNode
4412 Jps
4255 DataNode
命令執行成功
============= s103 jps ============
4256 JournalNode
2721 QuorumPeerMain
4328 Jps
4171 DataNode
命令執行成功
============= s104 jps ============
2707 QuorumPeerMain
4308 Jps
4151 DataNode
4236 JournalNode
命令執行成功
============= s105 jps ============
4388 DFSZKFailoverController
4284 NameNode
4446 Jps
命令執行成功
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ start-dfs.sh

2>.上傳測試數據到HDFS集群中

[yinzhengjie@s101 download]$ cat temp 
0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991902010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991903010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991904010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9-00611+99999101831ADDGF108991999999999999999999
0029029070999991905010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991906010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991907010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9-00671+99999101701ADDGF106991999999999999999999
0029029070999991908010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991909010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991910010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991911010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9-00611+99999101831ADDGF108991999999999999999999
0029029070999991912010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991913010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991914010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991915010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991916010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991917010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991918010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991919010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00281+99999101751ADDGF108991999999999999999999
0029029070999991920010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991902010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991903010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991904010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991905010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00561+99999101761ADDGF108991999999999999999999
0029029070999991906010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991907010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9-00671+99999101701ADDGF106991999999999999999999
0029029070999991908010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9+00781+99999102001ADDGF108991999999999999999999
0029029070999991909010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9-00721+99999102001ADDGF104991999999999999999999
0029029070999991910010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9+00941+99999102001ADDGF108991999999999999999999
0029029070999991911010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991912010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991913010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991914010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
0029029070999991915010113004+64333+023450FM-12+000599999V0202901N008219999999N0000001N9+00721+99999102001ADDGF104991999999999999999999
0029029070999991916010120004+64333+023450FM-12+000599999V0209991C000019999999N0000001N9-00941+99999102001ADDGF108991999999999999999999
0029029070999991917010206004+64333+023450FM-12+000599999V0201801N008219999999N0000001N9+00611+99999101831ADDGF108991999999999999999999
0029029070999991918010213004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9-00561+99999101761ADDGF108991999999999999999999
0029029070999991919010220004+64333+023450FM-12+000599999V0201801N009819999999N0000001N9+00281+99999101751ADDGF108991999999999999999999
0029029070999991920010306004+64333+023450FM-12+000599999V0202001N009819999999N0000001N9+00671+99999101701ADDGF1069919999999999999999990029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF108991999999999999999999
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ cat temp 技術分享圖片

[yinzhengjie@s101 download]$ hdfs dfs -mkdir -p  /home/yinzhengjie/data
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[yinzhengjie@s101 download]$ 
[yinzhengjie@s101 download]$ hdfs dfs -ls -R /
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/soft/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home/yinzhengjie
drwxr-xr-x   - yinzhengjie supergroup          0 2018-07-27 03:44 /home/yinzhengjie/data
[yinzhengjie@s101 download]$

[yinzhengjie@s101 download]$ hdfs dfs -mkdir -p /home/yinzhengjie/data

3>.

二.部署Spark集群

三.在Spark集群中執行代碼

Scala進階之路-Spark獨立模式集群部署

Scala進階之路-Spark獨立模式集群部署

Scala進階之路-Spark獨立模式集群部署

Scala進階之路-面向對象編程之類的成員詳解

慕課從零到一spark進階之路（一）

Esp8266進階之路17 esp8266自研的快速上電開關五次 (開-關為一次) ，無需按鍵觸發則8266進去一鍵配網模式。（附帶Demo）

CSS進階之路

進階之路（基礎篇） - 011 arduino api基礎手冊

進階之路（基礎篇） - 008 SPI數據傳輸(庫函數方法)

進階之路（基礎篇） - 009 通過底層AVR方法實現SPI數據傳輸

進階之路（基礎篇） - 007 脈沖寬度測量

進階之路（中級篇） - 018 基於arduino的簡易版智能衣架

【SSH進階之路】Hibernate映射——一對一單向關聯映射（五）

【SSH進階之路】Struts + Spring + Hibernate 進階開端（一）

2017PHP程序員的進階之路

樹莓派進階之路 (012) - 關於Raspberry Pi樹莓派無線網卡配置

【SSH進階之路】Hibernate基本映射（三）

GO語言的進階之路-go的程序結構以及包簡介

【SSH進階之路】Struts基本原理 + 實現簡單登錄（二）

樹莓派進階之路 (023) - Windows下用串行連接控制樹莓派（轉）

樹莓派進階之路 (024) - windows遠程桌面連接樹莓派通過xrdp服務（轉）

Java進階之路——從初級程序員到架構師，從小工到專家

Scala進階之路-Spark獨立模式集群部署

相關推薦