大數據平臺 CDH5.12.2 安裝部署
阿新 • • 發佈：2018-04-10
大數據 cdh hadoop
- 一：初始化環境部署配置
- 二： CDH 5.12.2 部署處理
一： 初始化環境部署處理
1.1 環境準備：
Centos7.3x64 所有主機名配置 配置好 所有機器的無密鑰登陸1.2 主機名配置
172.17.100.131 node01.yangyang.com node01 172.17.100.132 node02.yangyang.com node02 172.17.100.133 node03.yangyang.com node03 172.17.100.134 node04.yangyang.com node04 172.17.100.135 node05.yangyang.com node05 172.17.100.136 node06.yangyang.com node06
1.3 無密鑰登錄配置
做root用戶無密鑰認證
ssh-keygen ---一直敲回車到最後
cat id_rsa.pub >> authorized_keys
將所有的公鑰導入authorized_keys 分發到 所有的 機器的.ssh/ 下面
然後測試1.4 關閉 firewalld，iptables 與 selinux
yum install -y firewall* iptable* (1) 關閉firewalld systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --state (2) 關閉iptables iptables -F systemctl stop iptables.service service iptables save systemctl disable iptables.service (3) 禁用 selinux sed -i ‘/SELINUX/s/enforcing/disabled/‘ /etc/selinux/config setenforce 0 getenforce 0 sestatus 重啟機器
1.4 所有機器更改centos7.3x64的文件句柄數
vim /etc/security/limits.conf* soft nofile 65535 * hard nofile 1029345 * soft nproc unlimited * hard nproc unlimited * soft memlock unlimited * hard memlock unlimited
1.5 所有機器 配置好jdk環境 同步所有機器
(1) 卸載原有jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps
(2) 安裝jdk環境
rpm -ivh jdk-8u151-linux-x64.rpm1.6 生產環境配置chrony 時間同步服務器
所有機器 yum install chrony*
1. node01:
vim /etc/chrony.conf 增加
----
註釋掉相關默認的server
server ntp1.aliyun.com iburst
allow 172.17.100.0.0/16
service chronyd start
chkconfig chronyd on
2. node02---node05 同步node01
vim /etc/chrony.conf
----
註釋掉默認的server
server 172.17.100.131 iburst
----
service chronyd start
chkconfig chronyd on 1.7 安裝mysql5.7數據庫主節點上面安裝
卸載：mariadb-libs 的包：
----
＃rpm -e mariadb-libs mariadb-devel --nodeps
---
安裝mysql5.7 所需要依賴包
# yum install -y perl-JSON libaio
解壓 mysql 5.7.21 集成安裝包
tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar
刪掉一個包mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm 這個minimal包不然裝不了mysql5.7
# rm -rf mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm
# rpm -ivh mysql-community-*
# mysqld --initialize-insecure --user=mysql
註：上面的安裝步驟是用忽略安裝mysql包安裝順序，系統將自動識別安裝順序
啟動mysql5.7
#service mysqld start
# chkconfig mysqld on
註：初次使用mysql的時候,需要更改mysql 的密碼，否則查詢不了mysql數據庫當中的數據。
#mysql -uroot -p
mysql > set password = password(‘Bigdata_1234‘);
mysql> flush privileges;1.8 修改系統參數
vim /etc/sysctl.conf
##內核默認參數
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
##打開文件數參數(20*1024*1024)
fs.file-max= 20971520
##WEB Server參數
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1200
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 5000
##TCP補充參數
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
##禁用ipv6
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
##swap使用率優化
vm.swappiness=0
---
執行以下命令使參數生效:
# sysctl -p
同步到所有機器二： CDH5.12.2 安裝與配置
2.1 安裝CM 所依賴的包：(全部節點都安裝)
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python2.2 安裝cloudera-server 與更改配置 （節點全部安裝）
rpm -ivh cloudera-manager-*
rpm -ivh enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
在mysql 當中創建CM 的庫
# mysql -uroot -pBigdata_1234
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘%‘ identified by ‘cmf_1234‘ with grant option;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘node01.yangyang.com‘ identified by ‘cmf_1234‘ with grant option;
修改cm數據庫
#cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak
# vim /etc/cloudera-scm-server/db.properties 
2.3 安裝 離線cdh parcel-repo
#cd /root/software
#cp -ap CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1 (去掉 這個 1) CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ （所有主機都要執行這條命令）
2.4 jdbc 配置
拷貝jdbc 到 java的目錄
tar -zxvf mysql-connector-java-5.1.44.tar.gz
cd mysql-connector-java-5.1.44
cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/
cd /usr/share/java/
mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
2.5 修改config.ini 文件
#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak
#vim /etc/cloudera-scm-agent/config.ini
修改server_host 為主節點的IP 地址
---
server_host=172.17.100.131
---
同步到所有主機節點
scp /etc/cloudera-scm-agent/config.ini node02:/etc/cloudera-scm-agent/
scp /etc/cloudera-scm-agent/config.ini node03:/etc/cloudera-scm-agent/
2.6 啟動CM-server與agent-server
node01.yangyang.com 主機
cd /etc/init.d
./cloudera-scm-server start
查看啟動日誌
查看啟動結果：
需要通過查看server的啟動日誌文件
/var/log/cloudera-scm-server
tail -f cloudera-scm-server.log
看到jetty 表示啟動成功 會有 7180 端口啟動：
2.7打開web瀏覽器：
http://172.17.100.131:7180
用戶名： admin
密碼： admin
啟動 所有的agent 節點
cd /etc/init.d/
./cloudera-scm-agent start
一直到分發結束 -- 然後繼續
所有節點執行：
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
在最下面增加：
---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步所有機器
重新運行界面：
配置角色：
數據庫授權：
mysql -uroot -pBigdata_1234
---
創建hive 所需要的庫並授權：
mysql >create database cdh_hive default character set latin1;
字符必須為拉丁(latin1),如果uft8可能會報錯Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘%‘ identified by ‘Hive_123‘ with grant option;
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘node01.yangyang.com‘ identified by ‘Hive_123‘ with grant option;
mysql >flush privileges; 
創建oozie 所需要的庫
mysql >create database cdh_oozie default character set utf8;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘%‘ identified by ‘Oozie_123‘ with grant option;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘node01.yangyang.com‘ identified by ‘Oozie_123‘ with grant option;
mysql >flush privileges; 
創建hue所需要的庫：
mysql >create database cdh_hue default character set utf8;
mysql >grant all privileges on cdh_hue.* to ‘hue’@‘%‘ identified by ‘Hue_123‘ with grant option;
mysql >grant all privileges on cdh_hue.* to ‘hue‘@‘node01.yangyang.com‘ identified by ‘Hue_123‘ with grant option;
mysql >flush privileges; 
創建CM 的 監控端Activity Monitor 的庫
Activity Monitor:
mysql >create database cdh_am default character set utf8;
mysql >grant all privileges on cdh_am.* to ‘am’@‘%‘ identified by ‘Am_123‘ with grant option;
mysql >grant all privileges on cdh_am.* to ‘am‘@‘node01.yangyang.com‘ identified by ‘Am_123‘ with grant option;
mysql >flush privileges;
大數據平臺 CDH5.12.2 安裝部署