spark-2.4.0-hadoop2.7-安裝部署
1. 主機規劃
| 主機名稱 |
IP地址 |
作業系統 |
部署軟體 |
執行程序 |
備註 |
| mini01 |
172.16.1.11【內網】 10.0.0.11 【外網】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7【主】 |
QuorumPeerMain、 |
|
| mini02 |
172.16.1.12【內網】 10.0.0.12 【外網】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0 |
QuorumPeerMain、 |
|
| mini03 |
172.16.1.13【內網】 10.0.0.13 【外網】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
|
| mini04 |
172.16.1.14【內網】 10.0.0.14 【外網】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
|
| mini05 |
172.16.1.15【內網】 10.0.0.15 【外網】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
|
說明
該Spark叢集安裝，但是有一個很大的問題，那就是Master節點存在單點故障，要解決此問題，就要藉助zookeeper，並且啟動至少兩個Master節點來實現高可靠。具體部署下節講解。
2. 免密碼登入
實現mini01到mini02、mini03、mini04、mini05通過祕鑰免密碼登入。
參見文章：Hadoop2.7.6_01_部署
3. Jdk【java8】
參見文章：Hadoop2.7.6_01_部署
4. Spark部署步驟
4.1. Spark安裝
1 [[email protected] software]$ pwd 2 /app/software 3 [[email protected] software]$ ll 4 total 238572 5 -rw-r--r-- 1 yun yun 227893062 Nov 19 21:24 spark-2.4.0-bin-hadoop2.7.tgz 6 [[email protected] software]$ tar xf spark-2.4.0-bin-hadoop2.7.tgz 7 [[email protected] software]$ mv spark-2.4.0-bin-hadoop2.7 /app/ 8 [[email protected] software]$ cd /app/ 9 [[email protected] ~]$ ln -s spark-2.4.0-bin-hadoop2.7/ spark 10 [[email protected] ~]$ ll -d spark-* 11 drwxr-xr-x 13 yun yun 211 Oct 29 14:36 spark-2.4.0-bin-hadoop2.7 12 lrwxrwxrwx 1 yun yun 26 Nov 24 14:23 spark -> spark-2.4.0-bin-hadoop2.7/
4.2. 環境變數修改
根據規劃，該環境變數的修改包括mini01、mini03、mini04、mini05。
1 # 需要root許可權去新增環境變數 2 [[email protected] ~]# tail /etc/profile 3 ……………… 4 # spark環境變數 5 export SPARK_HOME="/app/spark" 6 export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH 7 8 [[email protected] ~]# logout 9 [[email protected] conf]$ source /etc/profile # 重新載入該環境變數
4.3. 配置修改
1 [[email protected] conf]$ pwd 2 /app/spark/conf 3 [[email protected] conf]$ cp -a spark-env.sh.template spark-env.sh 4 [[email protected] conf]$ tail spark-env.sh # 修改環境變數配置 5 # Options for native BLAS, like Intel MKL, OpenBLAS, and so on. 6 # You might get better performance to enable these options if using native BLAS (see SPARK-21305). 7 # - MKL_NUM_THREADS=1 Disable multi-threading of Intel MKL 8 # - OPENBLAS_NUM_THREADS=1 Disable multi-threading of OpenBLAS 9 10 # 新增配置如下 11 # 配置JAVA_HOME 12 export JAVA_HOME=/app/jdk 13 # 設定Master的主機名 14 export SPARK_MASTER_IP=mini01 15 # 每一個Worker最多可以使用的記憶體，我的虛擬機器就2g 16 # 真實伺服器如果有128G，你可以設定為100G 17 # 所以這裡設定為1024m或1g 18 export SPARK_WORKER_MEMORY=1024m 19 # 每一個Worker最多可以使用的cpu core的個數，我虛擬機器就一個... 20 # 真實伺服器如果有32個，你可以設定為32個 21 export SPARK_WORKER_CORES=1 22 # 提交Application的埠，預設就是這個，萬一要改呢，改這裡 23 export SPARK_MASTER_PORT=7077 24 25 [[email protected] conf]$ pwd 26 /app/spark/conf 27 [[email protected] conf]$ cp -a slaves.template slaves 28 [[email protected] conf]$ tail slaves # 修改slaves 配置 29 # distributed under the License is distributed on an "AS IS" BASIS, 30 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 31 # See the License for the specific language governing permissions and 32 # limitations under the License. 33 # 34 35 # A Spark Worker will be started on each of the machines listed below. 36 mini03 37 mini04 38 mini05
4.4. 分發到其他機器
分發到mini03、mini04和mini05
1 [[email protected] ~]$ scp -pr spark-2.4.0-bin-hadoop2.7/ [email protected]:/app # 拷貝到mini03 2 [[email protected] ~]$ scp -pr spark-2.4.0-bin-hadoop2.7/ [email protected]:/app # 拷貝到mini04 3 [[email protected] ~]$ scp -pr spark-2.4.0-bin-hadoop2.7/ [email protected]:/app # 拷貝到mini05
在mini03、mini04和mini05上操作
1 [[email protected] ~]$ pwd 2 /app 3 [[email protected] ~]$ ll -d spark-2.4.0-bin-hadoop2.7 4 drwxr-xr-x 13 yun yun 211 Oct 29 14:36 spark-2.4.0-bin-hadoop2.7 5 [[email protected] ~]$ ln -s spark-2.4.0-bin-hadoop2.7/ spark 6 [[email protected] ~]$ ll -d spark-* 7 drwxr-xr-x 13 yun yun 211 Oct 29 14:36 spark-2.4.0-bin-hadoop2.7 8 lrwxrwxrwx 1 yun yun 26 Nov 24 23:39 spark -> spark-2.4.0-bin-hadoop2.7/
4.5. 啟動spark
在mini01上操作
1 [[email protected] sbin]$ pwd 2 /app/spark/sbin 3 [[email protected] sbin]$ ./start-all.sh # 關閉使用 stop-all.sh 指令碼 4 starting org.apache.spark.deploy.master.Master, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.master.Master-1-mini01.out 5 mini03: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker-1-mini03.out 6 mini05: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker-1-mini05.out 7 mini04: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker-1-mini04.out 8 [[email protected] ~]$ 9 [[email protected] ~]$ jps # 檢視程序狀態 10 3103 Master 11 3183 Jps
mini03程序檢視
1 [[email protected] ~]$ jps 2 2387 Worker 3 2437 Jps
mini04程序檢視
1 [[email protected] ~]$ jps 2 2183 Jps 3 2125 Worker
mini05程序檢視
1 [[email protected] ~]$ jps 2 2212 Worker 3 2261 Jps
4.6. 瀏覽器訪問
1 http://mini01:8080/
