CentOS7+Hadoop2.7.2(HA高可用+Federation聯邦)+Hive1.2.1+Spark2.1.0 完全分散式叢集安裝
本文件主要記錄了Hadoop+Hive+Spark叢集安裝過程,並且對NameNode與ResourceManager進行了HA高可用配置,以及對NameNode的橫向擴充套件(Federation聯邦)
1VM網路配置
將子網IP設定為192.168.1.0:
將閘道器設定為192.168.1.2:
並禁止DHCP
當經過上面配置後,虛擬網絡卡8的IP會變成192.168.1.1:
(虛擬機器與物理機不在一個網段是沒有關係的)
2CentOS配置
2.1下載地址
下載不帶桌面的最小安裝版本
2.2啟用網絡卡
啟用網絡卡,並設定相關IP:
閘道器與DNS設定為上面虛擬網絡卡8中設定的閘道器即可
2.3SecureCRT
當網絡卡啟用後,就可以使用SecureCRT終端遠端連線Linux,這樣方便後續操作。如何連線這裡省略,
這裡連線上後簡單的進行下面設定:
2.4修改主機名
/etc/sysconfig/network
/etc/hostname
/etc/hosts
192.168.1.11 node1
192.168.1.12 node2
192.168.1.13 node3
192.168.1.14 node4
2.5yum代理上網
由於公司內部是代理上網,所以
yum代理的設定:vi /etc/yum.conf
再次執行yum,發現可以連網搜尋軟體包了:
2.6安裝ifconfig
2.7wget安裝與代理
安裝好wget後,在/etc目錄下就會產生wget配置檔案wgetrc,在這裡面可以配置wget代理:
[[email protected] ~]# vi /etc/wgetrc
2.8安裝VMware Tools
為了虛擬機器與主機時間同步,所以需要安裝VMWare Tools
[[email protected] opt]# yum -y install perl
[[email protected] ~]# mount /dev/cdrom /mnt
[[email protected] ~]# tar -zxvf /mnt/VMwareTools-9.6.1-1378637.tar.gz -C /root
[[email protected] ~]# umount /dev/cdrom
[[email protected] ~]# /root/vmware-tools-distrib/vmware-install.pl
[[email protected] ~]# rm -rf /root/vmware-tools-distrib
注:下面檔案共享與滑鼠拖放功能不要安裝,否則安裝過程會出問題:
[[email protected] ~]# chkconfig --list | grep vmware
vmware-tools 0:關 1:關 2:開 3:開 4:開 5:開 6:關
vmware-tools-thinprint 0:關 1:關 2:開 3:開 4:開 5:開 6:關
[[email protected] ~]# chkconfig vmware-tools-thinprint off
[[email protected] ~]# find / -name *vmware-tools-thinprint* | xargs rm -rf
2.9其他
2.9.1問題
剛啟動時會出以下錯誤提示:
修改虛擬機器配置檔案node1.vmx可以解決:
vcpu.hotadd = "FALSE"
mem.hotadd = "FALSE"
2.9.2設定
2.9.2.1去掉開機等待時間
[[email protected] ~]# vim /etc/default/grub
GRUB_TIMEOUT=0 #預設為5秒
[[email protected] ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
2.9.2.2VM調整
(注:小記憶體禁用)
修改node1.vmx檔案:
mainMem.useNamedFile = "FALSE"
為了全屏顯示,方便命令列輸入,做以下調整:
並去掉狀態列顯示:
2.9.3命令
2.9.3.1關機與重啟
[[email protected] ~]# reboot
[[email protected] ~]# shutdown -h now
2.9.3.2服務停止與禁用
#檢視開機自啟動服務
[[email protected] ~]# systemctl list-unit-files | grep enabled | sort
auditd.service enabled
crond.service enabled
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.nm-dispatcher.service enabled
default.target enabled
dm-event.socket enabled
[email protected] enabled
irqbalance.service enabled
lvm2-lvmetad.socket enabled
lvm2-lvmpolld.socket enabled
lvm2-monitor.service enabled
microcode.service enabled
multi-user.target enabled
NetworkManager-dispatcher.service enabled
NetworkManager.service enabled
postfix.service enabled
remote-fs.target enabled
rsyslog.service enabled
sshd.service enabled
systemd-readahead-collect.service enabled
systemd-readahead-drop.service enabled
systemd-readahead-replay.service enabled
tuned.service enabled
[[email protected] ~]# systemctl | grep running | sort
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
dbus.socket loaded active running D-Bus System Message Bus Socket
[email protected] loaded active running Getty on tty1
irqbalance.service loaded active running irqbalance daemon
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-lvmetad.socket loaded active running LVM2 metadata daemon socket
NetworkManager.service loaded active running Network Manager
polkit.service loaded active running Authorization Manager
postfix.service loaded active running Postfix Mail Transport Agent
rsyslog.service loaded active running System Logging Service
session-1.scope loaded active running Session 1 of user root
session-2.scope loaded active running Session 2 of user root
session-3.scope loaded active running Session 3 of user root
sshd.service loaded active running OpenSSH server daemon
systemd-journald.service loaded active running Journal Service
systemd-journald.socket loaded active running Journal Socket
systemd-logind.service loaded active running Login Service
systemd-udevd-control.socket loaded active running udev Control Socket
systemd-udevd-kernel.socket loaded active running udev Kernel Socket
systemd-udevd.service loaded active running udev Kernel Device Manager
tuned.service loaded active running Dynamic System Tuning Daemon
vmware-tools.service loaded active running SYSV: Manages the services needed to run VMware software
wpa_supplicant.service loaded active running WPA Supplicant daemon
#檢視一個服務的狀態
systemctl status auditd.service
#開機時啟用一個服務
systemctl enable auditd.service
#開機時關閉一個服務
systemctl disable auditd.service
systemctl disable postfix.service
systemctl disable rsyslog.service
systemctl disable wpa_supplicant.service
#檢視服務是否開機啟動
systemctl is-enabled auditd.service
2.9.3.3查大檔案目錄
find . -type f -size +10M -print0 | xargs -0 du -h | sort -nr
將前最大的前20目錄列出來,--max-depth表示目錄深度,如果去掉,則遍歷所有子目錄:
du -hm --max-depth=5 / | sort -nr | head -20
find /etc -name '*srm*'#表示在/etc目錄下查詢檔名中含有字元
2.9.3.4檢視磁碟使用情況
[[email protected] dev]# df -h
檔案系統容量已用可用 已用% 掛載點
/dev/mapper/centos-root 50G 1.5G 49G 3% /
devtmpfs 721M 0 721M 0% /dev
tmpfs 731M 0 731M 0% /dev/shm
tmpfs 731M 8.5M 723M 2% /run
tmpfs 731M 0 731M 0% /sys/fs/cgroup
/dev/mapper/centos-home 47G 33M 47G 1% /home
/dev/sda1 497M 106M 391M 22% /boot
tmpfs 147M 0 147M 0% /run/user/0
2.9.3.5檢視記憶體使用情況
[[email protected] dev]# top
3安裝JDK
線上下載jdk-8u72-linux-x64.tar.gz,並存放在/root下:
[[email protected] ~]# tar -zxvf /root/jdk-8u92-linux-x64.tar.gz -C /root
[[email protected] ~]# vi /etc/profile
在/etc/profile檔案的最末加上如下內容:
export JAVA_HOME=/root/jdk1.8.0_92
export PATH=.:$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[[email protected] ~]# source /etc/profile
[[email protected] ~]# java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
使用env命令檢視當前設定的環境變數是否正確:
[[email protected] ~]# env | grep CLASSPATH
CLASSPATH=.:/root/jdk1.8.0_92/jre/lib/rt.jar:/root/jdk1.8.0_92/lib/dt.jar:/root/jdk1.8.0_92/lib/tools.jar
4複製虛擬機器
前面只安裝一臺node1的物理機,現從node1複製出node2\node3\node3
node1 |
192.168.1.11 |
node2 |
192.168.1.12 |
node3 |
192.168.1.13 |
node4 |
192.168.1.14 |
修改相應虛擬機器顯示名:
開機時選擇複製:
修改主機名:
[[email protected] ~]# vi /etc/sysconfig/network
[[email protected] ~]# vi /etc/hostname
5SSH 免密碼登入
RSA加密演算法是一種典型的非對稱加密演算法
RSA演算法可以用於資料加密(公鑰加密,私鑰解密)和數字簽名或認證(私鑰加密,公鑰解密)
5.1一般的ssh原理(需要密碼)
客戶端向伺服器端發出連線請求
伺服器端向客戶端發出自己的公鑰
客戶端使用伺服器端的公鑰加密通訊登入密碼然後發給伺服器端
如果通訊過程被截獲,由於竊聽者即使獲知公鑰和經過公鑰加密的內容,但不擁有私鑰依然無法解密(RSA演算法)
伺服器端接收到密文後,用私鑰解密,獲知通訊密碼
5.2免密碼原理
先在客戶端建立一對密匙,並把公用密匙放在需要訪問的伺服器上
客戶端向伺服器發出請求,請求用你的密匙進行安全驗證
伺服器收到請求之後, 先在該伺服器上你的主目錄下尋找你的公用密匙,然後把它和你傳送過來的公用密匙進行比較。如果兩個密匙一致, 伺服器就用公用密匙加密“質詢”(challenge)並把它傳送給客戶端
客戶端收到“質詢”之後就可以用自己的私人密匙解密再把它傳送給伺服器
伺服器比較發來的“質詢”和原先的是否一致,如果一致則進行授權,完成建立會話的操作
5.3SSH免密碼
先刪除以前生成的:
rm -rf /root/.ssh
生成金鑰:
[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-keygen -t rsa
[[email protected] ~]# ssh-keygen -t rsa
命令“ssh-keygen -t rsa”表示使用 rsa 加密方式生成金鑰, 回車後,會提示三次輸入資訊,我們直接回車即可。
檢視生成的金鑰:
其中id_rsa.pub為公鑰,id_rsa為私鑰
伺服器之間公鑰拷貝:
ssh-copy-id -i /root/.ssh/id_rsa.pub <主機名>
表示將本機的公鑰拷貝到hadoop-slave1主機上去,並自動追加到authorized_keys檔案中去,如果不存在則會自動建立一個。如果是自己遠端自己時,主機就填自己
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node1
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node2
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node3
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node4
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node1
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node2
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node3
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node4
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node1
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node2
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node3
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node4
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node1
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node2
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node3
[[email protected] ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub node4
注:如果發現三臺虛擬機器上生成的公鑰都是一個時,請先刪除/etc/udev/rules.d/70-persistent-net.rules 檔案,再刪除 /root/.ssh資料夾後,重新生成
6HA+Federation伺服器規劃
node1 |
node2 |
node3 |
node4 |
||
NameNode |
Hadoop |
Y(屬於cluster1 |
Y叢集1) |
Y(屬於cluster2 |
Y叢集2) |
DateNode |
Y |
Y |
Y |
||
NodeManager |
Y |
Y |
Y |
||
JournalNodes |
Y |
Y |
Y |
||
zkfc(DFSZKFailoverController) |
Y(有namenode的地方 |
Y就有zkfc) |
Y |
Y |
|
ResourceManager |
Y |
Y |
|||
ZooKeeper(QuorumPeerMain) |
Zookeeper |
Y |
Y |
Y |
|
MySQL |
HIVE |
Y |
|||
metastore(RunJar) |
Y |
||||
HIVE(RunJar) |
Y |
||||
Scala |
Spark |
Y |
Y |
Y |
Y |
Spark-master |
Y |
||||
Spark-worker |
相關推薦no |