基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫01 —— 準備環境，搭建本地倉庫，安裝ambari

阿新 • • 發佈：2019-01-18

一、叢集軟硬體環境準備：

作業系統： centos 7 x86_64.1804

Ambari版本：2.7.0

HDP版本：3.0.0

HAWQ版本：2.3.0
5臺PC作為工作站：

ep-bd01

ep-bd02

ep-bd03

ep-bd04

ep-bd05

其中ep-bd01作為主節點，用於安裝ambari-server。

二、配置作業系統，安裝必備軟體

2，安裝配置NTP服務，保證叢集時間保持同步，以防止由於時間不同而造成掉線故障。

3，安裝MariaDB Server用於Ambari server以及Hue和Hive

4，安裝yum priorities plugin

yum install yum-plugin-priorities -y

三、搭建本地倉庫：

1，下載軟體包：

cd /root
mkdir downloads
cd downloads
wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo
wget 
 http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz 

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz

2、搭建本地倉庫：

安裝並開啟Apache HTTP服務

yum install httpd -y
systemctl enable httpd
systemctl start httpd

安裝HTTPD服務

確保/var/www/html目錄存在，沒有的話建立。

mkdir -p /var/www/html

建立HDP，HDF子目錄

cd /var/www/html
mkdir hdp  hdf

解開下載的軟體包：

cd /var/www/html
tar -zxvf  /root/downloads/ambari-2.7.0.0-centos7.tar.gz    -C .
tar -zxvf  /root/downloads/HDP-3.0.0.0-centos7-rpm.tar.gz   -C ./hdp
tar -zxvf  /root/downloads/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz  -C ./hdp
tar -zxvf  /root/downloads/HDP-UTILS-1.1.0.22-centos7.tar.gz   -C ./hdp

修改下載的ambari.repo，

vim ambari.repo

安裝如下內容修改，[注意版本號，需要根據具體下載的版本不同修改，解壓後自己檢視一下]：

#VERSION_NUMBER=2.7.0.0-897

[ambari-2.7.0.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.0.0
baseurl=http://ep-bd01/ambari/centos7/2.7.0.0-897
gpgcheck=1
gpgkey=http://ep-bd01/ambari/centos7/2.7.0.0-897/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

複製到/etc/yum.repos.d

cp ambari.repo   /etc/yum.repos.d/ambari.repo

修改下載的hdp.repo，

vim hdp.repo

安裝如下內容修改，[注意版本號，需要根據具體下載的版本不同修改，解壓後自己檢視一下]：

#VERSION_NUMBER=3.0.0.0-1634
[HDP-3.0]
name=HDP Version - HDP-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-3.0-GPL]
name=HDP GPL Version - HDP-GPL-3.0.0.0
baseurl=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

儲存退出，複製到/etc/yum.repos.d/

cp  hdp.repo  /etc/yum.repos.d/hdp.repo

四、主節點安裝ambari server

1，使用剛才配置好的本地倉庫，直接yum命令安裝。

yum install ambari-server -y

2，檢視ambari server 狀態

 systemctl status ambari-server 

● ambari-server.service - LSB: ambari-server daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

看到ambari server已成功安裝了。

3，配置mariadb，建立使用者和資料庫供ambari使用

建立資料庫使用者ambari

mysql -uroot -p

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> grant all privileges on *.* to 'ambari'@'%' identified by 'ambari';
Query OK, 0 rows affected (0.06 sec)

MariaDB [mysql]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [mysql]>

建立資料庫ambari，並執行ambari資料庫建表sql命令檔案。

MariaDB [mysql]> create database ambari;
Query OK, 1 row affected (0.01 sec)

MariaDB [mysql]> use ambari;
Database changed
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.06 sec)

Query OK, 0 rows affected (0.00 sec)
Statement prepared
... ...

4，配置ambari server

執行命令

ambari-server setup

回答選擇項，其中JDK選擇“”Custom“”，給出系統安裝目錄，資料庫一定要選擇高階配置，指定mariadb資料庫和使用者，本人系統中詳細過程如下：

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? ^C
Aborting ... Keyboard Interrupt.
[[email protected]-bd01 downloads]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: ^C
Aborting ... Keyboard Interrupt.
[[email protected]-bd01 downloads]# echo $JAVA_HOME
/usr/java/jdk1.8.0_181-amd64
[[email protected]-bd01 downloads]# $JAVA_HOME/bin/java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[[email protected]-bd01 downloads]# ambari-server setup         
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: ^C
Aborting ... Keyboard Interrupt.
[[email protected]-bd01 downloads]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n   
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
ERROR: Exiting with exit code 1. 
REASON: Downloading or installing JDK failed: 'Fatal exception:  Failed to download JDK: <urlopen error [Errno -2] Name or service not known>. Please check that the JDK is available at http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz. Also you may specify JDK file location in local filesystem using --jdk-location command line argument., exit code 1'. Exiting.
[[email protected]-bd01 downloads]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_181-amd64
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): ep-bd01
Port (3306): 
Database name (ambari): ambari
Username (ambari): ambari
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.0.0.897.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

5，命令列方式設定mysql資料庫連線庫，用於oozie和ranger連線mariadb時使用（上面設定後不起作用，必須如下操作）：

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

6，配置ambari-server自啟動，啟動ambari-server

[[email protected] downloads]# systemctl enable ambari-server
[[email protected]-bd01 downloads]# systemctl start ambari-server
[[email protected]-bd01 downloads]# systemctl status ambari-server
● ambari-server.service - LSB: ambari-server daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-14 11:06:16 CST; 2min 36s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 323056 ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/ambari-server.service
           └─323080 /usr/java/jdk1.8.0_181-amd64/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -...

Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Organizing resource files at /var/lib/ambari-server/resources...
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Ambari database consistency check started...
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server out at: /var/log/ambari-server/ambari-server.out
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server log at: /var/log/ambari-server/ambari-server.log
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Waiting for server start.....................
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Server started listening on 8080
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: DB configs consistency check: no errors and warnings were found.
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Ambari Server 'start' completed successfully.
Aug 14 11:06:16 ep-bd01 systemd[1]: Started LSB: ambari-server daemon.systemctl status ambari-server
● ambari-server.service - LSB: ambari-server daemon
   Loaded: loaded (/etc/rc.d/init.d/ambari-server; bad; vendor preset: disabled)
   Active: active (running) since Tue 2018-08-14 11:06:16 CST; 2min 36s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 323056 ExecStart=/etc/rc.d/init.d/ambari-server start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/ambari-server.service
           └─323080 /usr/java/jdk1.8.0_181-amd64/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -...

Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Organizing resource files at /var/lib/ambari-server/resources...
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Ambari database consistency check started...
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server PID at: /var/run/ambari-server/ambari-server.pid
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server out at: /var/log/ambari-server/ambari-server.out
Aug 14 11:05:58 ep-bd01 ambari-server[323056]: Server log at: /var/log/ambari-server/ambari-server.log
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Waiting for server start.....................
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Server started listening on 8080
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: DB configs consistency check: no errors and warnings were found.
Aug 14 11:06:16 ep-bd01 ambari-server[323056]: Ambari Server 'start' completed successfully.
Aug 14 11:06:16 ep-bd01 systemd[1]: Started LSB: ambari-server daemon.

ok，Ambari-server安裝完成。瀏覽器訪問ambari-server服務：

http://ep-bd01:8080

預設使用者名稱/密碼為：admin/admin，登陸後介面如下：

五、所有主機節點，安裝ambari-agent，並配置自啟動

yum install ambari-agent -y 
systemctl enable ambari-agent 
systemctl restart ambari-agent && systemctl status ambari-agent

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫01 —— 準備環境，搭建本地倉庫，安裝ambari

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ數據倉庫之一 —— MariaDB 安裝配置

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】HAWQ資料倉庫使用之 gpfdist協議

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫04 —— 安裝HAWQ外掛PXF3.3.0.0

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫02 ——使用ambari-server安裝HDP

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫01 —— 準備環境，搭建本地倉庫，安裝ambari

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫——安裝配置OPEN-SSH，設定主機節點之間免密互訪

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫——作業系統配置，安裝必備軟體

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫03 —— 安裝HAWQ 2.3.0.0

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫——安裝配置NTP服務，保證叢集時間保持同步

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫 —— MariaDB 安裝配置

【CentOS-7+ Ambari 2.7.0 + HDP 3.0+HAWQ2.3.00】遭遇問題及解決記錄

Centos升級Python 2.7並安裝pip、ipython

基於ambari 2.4.1 HDP-2.5.0 構建安全的hive資料庫

ambari 2.7 編譯與安裝

編譯 ambari 2.7.3

File Cabinet Pro for Mac(Mac選單欄檔案管理器) V6.7.1(2.7.9)破解版

MTK phonebook vCard 2 1 和vCard 3 0 有何不同

註解框架AndroidAnnotations4.5.2在android Studio 3.0之後的配置

Jedis 2.9.1、2.10.0 與 3.0.0 釋出，Redis 的 Java 客戶端

升級還是權謀？從USB PD 2.0到3.0

基於【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ資料倉庫01 —— 準備環境，搭建本地倉庫，安裝ambari

相關推薦