CDH6.0.1詳細安裝步驟
CDH6.0.1詳細安裝步驟
系統環境
• 作業系統：3臺 CentOS 7.3 x64
• Cloudera Manager：6.0.1
• CDH: 6.0.1
| VM Name | VM IP Adress | Centos_HostName | Role |
|---|---|---|---|
| Centos_wjg1 | 192.168.1.101 | hadoop01 | Cloudera Manager Server + Cloudera Manager Agent |
| Centos_wjg2 | 192.168.1.102 | hadoop02 | Cloudera Manager Agent |
| Centos_wjg6 | 192.168.1.106 | hadoop06 | Cloudera Manager Agent |
一、配置網路環境
- 編輯/etc/sysconfig/network-scripts/ifcfg-eth0配置檔案，修改其中以下紅色顏色標示的專案：
vi /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eth0
UUID=3719cf03-5d05-454b-903f-e1e26ad34fd6
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.1.101
NETWORK=192.168.1.1
NETSTAT=255.255.255.0
GATEWAY=192.168.1.1
DNS1=222.186.160.70
DNS2=114.114.114.114
2.編輯/etc/default/grub檔案，在GRUB_CMD_LINE_LINUX=""項中，插入"net.ifnames=0 biosdevname=0"（與原有項之間用空格隔開），儲存退出。
3.呼叫命令grub2-mkconfig -o /boot/grub2/grub.cfg更新配置，然後直接reboot重啟。
這樣之後再用ifconfig -a命令檢視時，就有eth0網絡卡存在了。 - 更改hostname
hostnamectl set-hostname hadoop01 - 修改hosts檔案
vi /etc/hosts
192.168.1.101 hadoop01
192.168.1.102 hadoop02
192.168.1.106 hadoop06
-
關閉防火牆
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
systemctl stop firewalld.service
systemctl disable firewalld.service - 配置免密登入：
在每臺主機上生成rsa公鑰和私鑰
ssh-keygen -t rsa
cd .ssh
cat id_rsa.pub >> authorized_keys
登入其他主機，將其他主機的公鑰檔案內容都拷貝到hadoop01主機上的authorized_keys檔案中，命令如下：
[[email protected] .ssh]# ssh-copy-id -i hadoop01
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'hadoop01'"
and check to make sure that only the key(s) you wanted were added.
在每臺主機上重複以上步驟，使任意兩臺主機間都能相互連通。
至此，免密碼登入已經設定完成。
- 更改系統時間：
進入home目錄，修改.bash_profile
#setup time region
TZ='Asia/Shanghai'
export TZ
或者: tzselect
將當前時間和日期寫入BIOS，避免重啟後失效
hwclock –w
- 配置NTP同步各臺伺服器的時間
9.1主節點配置
在配置之前，先使用ntpdate手動同步一下時間，免得本機與對時中心時間差距太大，使得ntpd不能正常同步。這裡選用65.55.56.206作為對時中心,ntpdate -u ntp1.aliyun.com
vi /etc/ntp.conf
#Hosts on local network are less restricted.
取消下面一行的註釋，並將其中IP和掩碼修改為真實環境IP和掩碼，此行配置為允許ntp客戶端連線的配置
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
找到server 0.centos.pool.ntp.org iburst，並將所有server配置進行註釋
新增下面兩行內容
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
啟動NTP服務
設定後，重啟ntpd服務，用ntpstat來檢查效果【大概幾秒鐘就可以看到效果】
#systemctl restart ntpd
#ntpstat
vi /etc/sysconfig/ntpdate
#Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=yes
9.2 Agent節點配置
修改/etc/ntp.conf
對該檔案進行以下內容的修改：
1、註釋所有restrict和server配置
2、新增下面註釋，需要修改以下的IP為NTP伺服器的IP（在本案例中就是server主機的IP）
server 192.168.1.100
將所有主機的ntp服務設定為開機自動啟動
chkconfig ntpd on
systemctl enable ntpd.service
二、 配置163 yum源
1）下載repo檔案
cd /etc/yum.repos.d/
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
2）備份並替換系統的repo檔案
mv CentOS-Base.repo CentOS-Base.repo.bak
cp CentOS7-Base-163.repo CentOS-Base.repo
3）執行yum源更新命令
yum clean all
yum makecache
yum update
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb –y
yum install httpd mod_ssl openssl-devel python-psycopg2 –y
yum install MySQL-python
yum install perl
yum install autoconf
三、安裝MySQL:
檢視並解除安裝CentOS系統自帶的MySQL和Mariadb
rpm -qa | grep MySQL
rpm -qa | grep mariadb
rpm -e --nodeps MySQL-python-1.2.5-1.el7.x86_64
rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
下載並上傳以下rpm包到主節點
MySQL-client-5.6.42-1.el7.x86_64.rpm
MySQL-devel-5.6.42-1.el7.x86_64.rpm
MySQL-server-5.6.42-1.el7.x86_64.rpm
在主節點上執行以下命令安裝MySQL
rpm -ivh MySQL*
安裝完成後啟動MySQL:
service mysql start
systemctl start mysql.service
修改MySQL的配置檔案
vi /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
#Disabling symbolic-links is recommended to prevent assorted security risks;
#to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
#InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
獲取系統初始密碼：
[[email protected] wjg]# cat /root/.mysql_secret
#The random password set for the root user at Tue Dec 18 20:37:59 2018 (local time): pP4xR06U9nPyWrhA
登入MySQL:
[[email protected] wjg]# mysql -uroot -ppP4xR06U9nPyWrhA
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.42
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('xxxxxxx')
grant all on . to [email protected]"%" identified by " xxxxxxx "
設定MySQL開機自啟動
systemctl enable mysql.service
初始化MySQL
/usr/bin/mysql_secure_installation
下載mysql-connector-java-5.1.46.tar.gz並解壓
在解壓目錄下找到mysql-connector-java.jar, 複製到所有主機的/usr/share/java目錄下
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar
[[email protected] usr]# ls -l /usr/share/java/mysql-connector-java.jar
-rw-r--r-- 1 root root 1004838 Dec 19 20:45 /usr/share/java/mysql-connector-java.jar
下載Oracle JDK包oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm上傳到所有主機並安裝
下載地址：https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
Creating Databases for Cloudera Software
需要建的庫有
scm、amon、rman、hue、metastore、sentry、nav、navms、oozie
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm. TO 'scm'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman. TO 'rman'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue. TO 'hue'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore. TO 'metastore'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry. TO 'sentry'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav. TO 'nav'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms. TO 'navms'@'%' IDENTIFIED BY '[email protected]';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie. TO 'oozie'@'%' IDENTIFIED BY '[email protected]';
初始化cloudera manager資料庫
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
四、下載並安裝Cloudera Manager
https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
- cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
- cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
- cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
在主節點hadoop01伺服器上安裝1,2,3；
在Agent伺服器上安裝1,3；
在主節點hadoop01上執行：
systemctl start cloudera-scm-server
systemctl restart cloudera-scm-server
檢視日誌：
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
顯示如下表示服務已經正常起來
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
Cloudera Manager將自動下載檔案到/opt/cloudera/parcel-repo目錄下
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.torrent
manifest.json
在所有Agent節點hadoop01, hadoop02, hadoop06上執行：
配置Agent的Server指向
vi /etc/cloudera-scm-agent/config.ini
[General]
#Hostname of the CM server.
server_host=hadoop01
[Security]
#Use TLS and certificate validation when connecting to the CM server.
use_tls=1
啟動cloudera-scm-agent服務：
systemctl status cloudera-scm-agent
systemctl start cloudera-scm-agent
systemctl restart cloudera-scm-agent
Agent Log檔案位置：/var/log/cloudera-scm-agent
cloudera-scm-agent.log
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log
登入Cloudera Manager管理介面
使用者名稱和密碼都是admin
Cloudera 安裝Parcel提示主機執行狀況不良
解決方法
刪除agent目錄下面的cm_guid檔案，並重啟失敗節點的agent服務。
[[email protected] ~]# cd /var/lib/cloudera-scm-agent/
[[email protected] cloudera-scm-agent]# ll
total 32
-rw------- 1 root root 2 Dec 28 14:27 active_parcels.json
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 28 23:01 agent-cert
-rw-r--r-- 1 root root 36 Dec 28 14:25 cm_guid
-rw------- 1 root root 13813 Dec 29 07:10 response.avro
-rw-r--r-- 1 root root 36 Dec 28 14:25 uuid
[[email protected] cloudera-scm-agent]# rm -rf cm_guid
[[email protected] cloudera-scm-agent]# systemctl restart cloudera-scm-agent
在所有節點上刪除cm_guid以後，cloudera manager會自動繼續進行安裝。
稍微等一段時間後就會看到parcel已經安裝到所有主機上並全部啟用。
點選繼續就會進入檢查主機正確性的介面
第1個警告：Cloudera 建議將 /proc/sys/vm/swappiness 設定為最大值 10。當前設定為 30。使用 sysctl 命令在執行時更改該設定並編輯 /etc/sysctl.conf，以在重啟後儲存該設定。您可以繼續進行安裝，但 Cloudera Manager 可能會報告您的主機由於交換而執行狀況不良。以下主機將受到影響：
檢視詳細資訊
hadoop[01-02, 06]
解決辦法：
臨時調整： sysctl vm.swappiness=10
永久調整： vi /etc/sysctl.conf
增加下面一行
vm.swappiness=10
第2個警告：已啟用透明大頁面壓縮，可能會導致重大效能問題。請執行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此設定，然後將同一命令新增到 /etc/rc.local 等初始化指令碼中，以便在系統重啟時予以設定。以下主機將受到影響:
檢視詳細資訊
hadoop[01-02, 06]
解決辦法：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
修改/etc/rc.local將上面兩行新增進去
vi /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
第3個警告：psycopg2版本太低的處理
安裝pip
yum -y install epel-release
yum -y install python-pip
pip install --upgrade pip
yum install postgresql-devel
安裝psycopg2
下載psycopg2-2.7.6.1.tar.gz並解壓
[[email protected] psycopg2-2.7.6.1]# python setup.py build
[[email protected] psycopg2-2.7.6.1]# python setup.py install
重新執行檢查主機，所有專案檢查通過
點選繼續進入選擇安裝產品介面
選擇所有服務然後點選繼續，進入主機角色分配介面
點選按主機檢視對主機分配的角色會看得更清楚一些
點選繼續進入資料庫設定介面，填寫先前建立的資料庫名稱、使用者名稱和密碼
點選繼續進入稽核更改介面
點選繼續開始進行安裝
安裝完成後點選繼續，進入Summary介面
