大資料平臺HDP搭建

資料庫大資料 ApacheAmbari · 發表 2019-02-11 14:14:28

摘要： Apache Ambari是一個基於Web的支援Apache Hadoop叢集的供應、管理和監控的開源工具，Ambari已支援大多數Hadoop元件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI進行視覺化...

Apache Ambari是一個基於Web的支援Apache Hadoop叢集的供應、管理和監控的開源工具，Ambari已支援大多數Hadoop元件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI進行視覺化的叢集管理，簡化了大資料平臺的安裝、使用難度。

二、安裝部署

2.1 主機規劃

序號	IP地址	主機名	系統版本
1	172.20.2.222	ambari-server	centos7.3
2	172.20.2.203	hadoop-1	centos7.3
3	172.20.2.204	hadoop-2	centos7.3
4	172.20.2.205	hadoop-3	centos7.3

2.2 部署

2.2.1 基礎環境部署

a.修改主機名配置hosts

systemctl stop firewalld
hostnamectl set-hostnameambari-server#更改個主機名
sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
setenforce 0
cat >>/etc/hosts<<EOF#新增各個節點hosts解析
172.20.2.222ambari-server
172.20.2.203hadoop-1
172.20.2.204hadoop-2
172.20.2.205hadoop-3
EOF
複製程式碼

b.配置ambari-server免金鑰登入其他節點

ssh-keygen -t rsa#在ambari-server生成金鑰對
for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done
複製程式碼

c.在ambari-server安裝資料庫

在ambari的setup中我們可以選擇使用預設的postgresql，也可以自定義使用其他資料庫，此處選用mariadb，便於後期管理維護

yum -y install mariadb-server mysql-connector-java
systemctl start mariadb
mysql -uroot password "mysqladmin"
登入資料庫後我們採用root登入
複製程式碼

d.配置本地yum源

由於ambari的各個元件包很大,HDP大約快6G，我們採取離線安裝方式，將ambari-server配置為yum源，事先在網上下載HDP、HDP-UTILS和Ambari 軟體版本：

參考url：ambari各版本參考連結 HDP和HDP UTILS參考連結

ambari	HDP	HDP-UTILS
2.4.1.0	2.5.0.0	1.1.0.21

yum install httpd -y && systemctl start httpd
mkdir /software && cd /software
#下載軟體
wget -c http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.1.0/ambari-2.4.1.0-centos7.tar.gz
wget -c http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0/HDP-2.5.0.0-centos7-rpm.tar.gz
wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
#建立目錄
mkdir -p /var/www/html/ambari
mkdir /var/www/html/hdp/{HDP,HDP-UTILS-1.1.0.21}
#解壓軟體
tar -zxvf ambari-2.4.1.0-centos7.tar.gz -C /var/www/html/ambari/
tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.21/
tar -zxvf HDP-2.5.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
配置ambari YUM源
cat >/etc/yum.repos.d/ambari.repo<<EOF
#VERSION_NUMBER=2.4.1.0-22
[Updates-ambari-2.4.1.0]
name=ambari-2.4.1.0 - Updates
baseurl=http://172.20.2.222/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22/
gpgcheck=1
gpgkey=http://172.20.2.222/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
#配置hdpYUM源
cat >/etc/yum.repos.d/hdp.repo<<EOF
#VERSION_NUMBER=2.5.0.0-1245
[HDP-2.5.0.0]
name=HDP Version - HDP-2.5.0.0
baseurl=http://172.20.2.222/hdp/HDP/centos7/
gpgcheck=1
gpgkey=http://172.20.2.222/hdp/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.21
baseurl=http://172.20.2.222/hdp/HDP-UTILS-1.1.0.21/
gpgcheck=1
gpgkey=http://172.20.2.222/hdp/HDP-UTILS-1.1.0.21/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
EOF
複製程式碼

2.2.2 安裝配置ambari

a.安裝ambari server

yum clean all
yum makecache
yum -y install ambari-server
#進入資料庫建立ambari庫和匯入資料
mysql>create database ambari DEFAULT CHARSET utf8;
mysql>use ambari
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
複製程式碼

b.配置ambari

[root@ambari-server hdp]# ambari-server setup
Using python/usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.or
acle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u77-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u77
-linux-x64.tar.gzjdk-8u77-linux-x64.tar.gz... 100% (173.0 MB of 173.0 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u77-linux-x64.tar.gz
Installing JDK to /usr/jdk64/
Successfully installed JDK to /usr/jdk64/
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jc
e_policy-8.zip
Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip
Installing JCE policy...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (ambari-server): ambari-server
Port (3306): 
Database name (ambari): 
Username (root): root
Enter Database Password (mysqladmin): 
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r
esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
#啟動服務
ambari-server start
複製程式碼

c.瀏覽器文字介面安裝

瀏覽器訪問： http://172.20.2.222:8080 預設使用者名稱密碼admin/admin