基於hortonworks的大數據集群環境部署流水

阿新 • • 發佈：2018-10-05

3.x crypto oca 訪問控制 bzip2 ber apache服務器 -h anywhere

一、ambari及HDP安裝文件下載地址：

1、系統

操作系統：CentOS7

2、軟件

本次安裝采用最新版本：

ambari-2.7.0.0

hdp-3.0.0.0

詳細信息及下載地址如下：

Ambari 2.7.0 Repositories
Base URL ：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0
Repo File ：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo
Tarball md5 | asc ：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz

HDP 3.0 Repositories
Base URL ：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0
Repo File ：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo
Tarball md5 | asc ：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz
HDP-UTILS Base URL ：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7
Tarball md5 | asc ：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
HDP-GPL URL ：http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/hdp.gpl.repo
Tarball md5 | asc ：http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz

二、配置服務器

1、安裝必要依賴：

yum -y install net-tools vim ntp bzip2 gcc readline-devel zlib-devel httpd wget

2、所有服務器分別配置/etc/hosts文件：向其中添加以下3行
192.168.0.105 chavin.king chavin
192.168.0.107 chavin02.king chavin02
192.168.0.108 chavin03.king chavin03
192.168.0.109 chavin01.king chavin01

3、所有服務器均創建安裝用戶： ambari
groupadd ambari
useradd -g ambari ambari
echo "ambari" | passwd --stdin ambari

4、所有服務器為cloudera用戶配置sudo權限
chmod u+w /etc/sudoers
echo "ambari ALL=(root)NOPASSWD:ALL" >> /etc/sudoers
chmod u-w /etc/sudoers

5、所有服務器關閉防火墻、禁用selinux
sed -i ‘/SELINUX=enforcing/d‘ /etc/selinux/config
sed -i ‘/SELINUX=disabled/d‘ /etc/selinux/config
echo "SELINUX=disabled" >> /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld

6、所有服務器設置文件打開數量及最大進程數
cp /etc/security/limits.conf /etc/security/limits.conf.bak
echo "* soft nproc 32000" >>/etc/security/limits.conf
echo "* hard nproc 32000" >>/etc/security/limits.conf
echo "* soft nofile 65535" >>/etc/security/limits.conf
echo "* hard nofile 65535" >>/etc/security/limits.conf

7、配置SSH無秘鑰登錄：至少在namenode和resourcemanager節點配置
ssh-keygen -t rsa
ssh-copy-id chavin.king
ssh-copy-id chavin01.king
ssh-copy-id chavin02.king
ssh-copy-id chavin03.king

8、配置集群時間同步服務：通過ntp服務配置

ntp主節點配置：
cp /etc/ntp.conf /etc/ntp.conf.bak
cp /etc/sysconfig/ntpd /etc/sysconfig/ntpd.bak
echo "restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf
echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpd
systemctl restart ntpd

ntp客戶端配置：
#crontab -e
輸入如下一行內容，保存退出：
0-59/1 * * * * /opt/scripts/sync_time.sh
上述腳本內容如下：
# cat /opt/scripts/sync_time.sh
/sbin/service ntpd stop
/usr/sbin/ntpdate chavin
/sbin/service ntpd start

9、所有節點安裝java軟件並配置環境變量
export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH

10、安裝postgresql數據庫

參考文檔：https://www.cnblogs.com/wcwen1990/p/6655438.html

官網下載：postgresql-9.5.5.tar.bz2

安裝：
tar -jxvf postgresql-9.5.5.tar.bz2
cd postgresql-9.5.5
./configure --prefix=/pg955
make world
make install-world
userdel postgres
groupadd postgres
useradd -g postgres postgres
chown -R postgres:postgres /pg955/
su - postgres

配置環境變量：
vim .bash_profile
export PGHOME=/pg955
export PGDATA=/pg955/data
export LD_LIBRARY_PATH=$PGHOME/lib:$LD_LIBRARY_PATH
PATH=$PGHOME/bin:$PATH

初始化數據庫：

pg_ctl init

修改配置文件：

vim postgresql.conf

listen_addresses = ‘*‘

修改訪問控制文件：

vim pg_hba.conf

host all all 192.168.0.0/24 md5

啟動數據庫：

pg_ctl start -l $PGHOME/logfile 2>&1 >/dev/null

修改postgres數據庫密碼：

psql

postgres=# \password

創建ambari數據庫：

create user ambari with password ‘ambari‘;

create database ambari owner=ambari encoding ‘utf8‘ template template0;

grant all privileges on database ambari to ambari;

create schema ambari;

11、安裝apache服務器：

service httpd start

cd /var/www/htmp

mkdir ambari

mkdir hdp

三、安裝ambari

1、上傳文件並解壓相應文件：

上傳ambari和hdp安裝文件到主服務器上，解壓ambari文件到/var/www/htmp/ambari/目錄下，解壓hdp和hdp-untils文件到/var/www/htmp/hdp/目錄下。
tar -zxvf ambari-2.7.0.0-centos7.tar.gz -C /var/www/html/ambari/
tar -zxvf HDP-3.0.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp/

2、配置yum源文件
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo .
vim ambari.repo
#VERSION_NUMBER=2.7.0.0-897
[ambari-2.7.0.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.0.0
baseurl=http://chavin.king/ambari/ambari/centos7/2.7.0.0-897
gpgcheck=1
gpgkey=http://chavin.king/ambari/ambari/centos7/2.7.0.0-897/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo .
vim hdp.repo
#VERSION_NUMBER=3.0.0.0-1634
[HDP-3.0.0.0]
name=HDP Version - HDP-3.0.0.0
baseurl=http://chavin.king/hdp/HDP/centos7/3.0.0.0-1634
gpgcheck=1
gpgkey=http://chavin.king/hdp/HDP/centos7/3.0.0.0-1634/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://chavin.king/hdp/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://chavin.king/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

3、安裝ambari-server

yum -y install ambari-server

4、配置ambari-server
[root@chavin ~]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is ‘disabled‘
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
====================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_181
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
====================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
====================================================================
Enter choice (1): 4
Hostname (localhost): chavin.king
Port (5432):
Database name (ambari):
Postgres schema (ambari):
Username (ambari):
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
ambari-admin-2.7.0.0.897.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server ‘setup‘ completed successfully.

5、執行數據庫腳本

psql --host=chavin.king --port=5432 --username=ambari -f /var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql

6、啟動ambari-server
[root@chavin ~]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........................................
Server started listening on 8080
DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details.
Ambari Server ‘start‘ completed successfully.

通過 http://chavin.king:8080 地址登陸，可以看到如下登陸界面，初始用戶名和密碼都是admin。

技術分享圖片