Cloudera Manager大數據平臺搭建

阿新 • • 發佈：2018-06-13

CM Hadoop

    在企業裏快速搭建大數據平臺除了Ambria外還可以用Cloudera Manager，這兩種工具相對於純手工搭建確實便捷很多，但是有利也有弊，相對於新手來說就不便於掌握內部原理，不好排錯等,而純手工搭建，出錯較多，不容易成功。

一. 規劃：
192.168.3.201 server/agent mysql
192.168.3.202 agent namenode
192.168.3.203 agent namenode（備） resourcemanager

192.168.3.204 agent resourcemanager （備）
192.168.3.205 agent

二. 環境準備

系統：Centos 6.5 64位

配置IP，hostname,
各個服務器IP配置：# vim /etc/sysconfig/network-script/ifcfg-eth0
service network restart
各個服務器名配置：# vim /etc/sysconfig/network

vim /etc/hosts

            192.168.3.201    rac201.com
            192.168.3.202    rac202.com
            192.168.3.203    rac203.com
            192.168.3.204    rac204.com
            192.168.3.205    rac205.com

  3.檢查DNS配置，確保能上網
    # vim /etc/resolve.conf  
    #   ping www.baidu.com

  4.關閉防火墻和禁用selinux
     getenforce   查看selinux的狀態並修改為disable狀態
    #  vim  /etc/sysconfig/selinux
                 SELINUX=disable
            重啟生效

  5. 禁用和關閉防火墻
      臨時禁用  #  iptables -F    
                #  setenforce 0
            或設置開機不啟動
                #   service  iptables stop
                #   chkconfig iptables off

  6. 所有服務器卸載自帶的JDK，安裝新版本，比如1.7或1.8版本的
     # rpm  -qa | grep java
     # rpm -e --nodeps tzdata-java-2012j-1.el6.noarch java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
     # mkdir -p /opt/modules    
     # tar zxf jdk-8u141-linux-x64.gz -C /opt/modules   

  7. 對所有服務器配置JAVA環境變量
     # vi /etc/profile
     export JAVA_HOME=/opt/modules/jdk1.8.0_141
     export PATH=$PATH:$JAVA_HOME/bin
     # source /etc/profile
  8. 拷貝配置到其他服務器
     # scp -r  /opt/modules/jdk1.8.0_141  [email protected]:/opt/modules/
     # scp -r  /opt/modules/jdk1.8.0_141  [email protected]:/opt/modules/
     # scp -r  /opt/modules/jdk1.8.0_141  [email protected]:/opt/modules/
     # scp -r  /opt/modules/jdk1.8.0_141  [email protected]:/opt/modules/

     檢查：  java -version

  9. 配置服務器之間免密鑰登錄
     9.1  #  ssh-keygen
          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac201.com
          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac202.com
          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac203.com
          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac204.com
          #  ssh-copy-id -i ~/.ssh/id_rsa.pub rac205.com

     9.2  # ssh rac01.com
          # exit            #不要忘記連接後要退出
          # ssh rac02.com
          # exit
          # ssh rac03.com
          # exit
          # ssh rac04.com
          # exit
          # ssh rac05.com
          # exit

 10. 服務器時間同步
    10.1 啟動ntpd進程服務
        # service ntpd start
        # chkconfig ntpd on
        # chkconfig --list | grep ntpd      
    10.2 修改時間服務器的配置
        # vim /etc/ntp.conf
        打開註釋，修改成本服務器的網段
        # restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap               
        註釋掉以下幾行，可能是三行也可能是五行
        #server 0.centos.pool.ntp.org
        #server 1.centos.pool.ntp.org
        #server 2.centos.pool.ntp.org

        手工添加以下兩行
        server  127.127.1.0     # local clock
        fudge   127.127.1.0 s   
        啟動並且配置開機時啟動
        # service ntpd start && chkconfig ntpd on && chkconfig --list | grep ntpd

11. 對所有服務器設置能打開的文件最大數量、進程最大數，能使用的內存等
     # vi /etc/security/limits.conf             
        *               soft    nofile          32728
        *               hard    nofile          1024567
        *               soft    nproc           65535
        *               hard    nproc           unlimited
        *               soft    memlock         unlimited
        *               hard    memlock         unlimited

     復制limits.conf到其他服務器中
     # scp /etc/security/limits.conf [email protected]:/etc/security/limits.conf
     # scp /etc/security/limits.conf [email protected]:/etc/security/limits.conf
     # scp /etc/security/limits.conf [email protected]:/etc/security/limits.conf
     # scp /etc/security/limits.conf [email protected]:/etc/security/limits.conf
     重啟生效：  ulimit -a

三. Mysql數據庫安裝

在rac201服務器檢查並卸載系統自動安裝的Mysql版本,

rpm -qa | grep mysql
```
    # rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
```

采用yum源方式安裝Mysql5.6, 先下載yum源文件,

wget http://repo.mysql.com/mysql57-community-release-el6-8.noarch.rpm

    # rpm -Uvh mysql57-community-release-el6-8.noarch.rpm
    修改yum源文件：
      vim /etc/yum.repos.d/mysql-community.repo
      [mysql56-community]
      enabled=1
      gpgcheck=1

      [mysql57-community]
      enabled=0
      gpgcheck=1

      vim /etc/yum.repos.d/mysql-community-source.repo
      [mysql56-community-source]
      enabled=1   
      gpgcheck=1

      [mysql-tools-preview-source]
      name=MySQL Tools Preview - Source
      baseurl=http://repo.mysql.com/yum/mysql-tools-preview/el/6/SRPMS
      enabled=1   
      gpgcheck=1

    用yum repolist mysql這個命令查看一下是否已經有mysql可安裝文件
      #yum repolist all | grep mysql

    yum安裝mysql, 一直yes下去
    # yum -y install mysql-community-server

    安裝成功後，啟動並初始化
    # service mysqld start
    # service mysqld status

    mysql剛安裝完，mysql的root用戶的密碼默認是空的，需要給root用戶設置密碼密碼，執行下列命令進行安全設置
    # mysql_secure_installation        

    輸入當前用戶：回車

    輸入root密碼：123456

    production environment.

    Remove anonymous users? [Y/n] y
     ... Success!

    Normally, root should only be allowed to connect from ‘localhost‘.  This
    ensures that someone cannot guess at the root password from the network.

    Disallow root login remotely? [Y/n] n
     ... skipping.

    By default, MySQL comes with a database named ‘test‘ that anyone can
    access.  This is also intended only for testing, and should be removed
    before moving into a production environment.

    Remove test database and access to it? [Y/n] n
     ... skipping.

    Reloading the privilege tables will ensure that all changes made so far
    will take effect immediately.

    Reload privilege tables now? [Y/n] y
     ... Success!

四. Cloudera Manager安裝

在每臺服務器上安裝第三方依賴包，或者在Centos系統安裝時安裝好開發包
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
提前下載好cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.3.6_x86_64.tar.gz
在rac201服務器上創建CM的解壓目錄

mkdir -p /opt/cloudera-manager

tar -zxf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz -C /opt/cloudera-manager

在rac201服務器上修改修改agent配置

vi /opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent/config.ini

Hostname of the CM server.
```
server_host=rac201.com 
server_port=7182 
```

拷貝解壓後的CM文件夾到其他服務器

scp -r /opt/cloudera-manager/ [email protected]:/opt/

# scp -r /opt/cloudera-manager/ [email protected]:/opt/
# scp -r /opt/cloudera-manager/ [email protected]:/opt/
# scp -r /opt/cloudera-manager/ [email protected]:/opt/

在所有服務器上創建CM的用戶

useradd --system --home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server --no-create-home --shell=/bin/false cloudera-scm
```
# cat /etc/passwd | grep cloudera-scm
```

設置CM數據庫的權限

mysql -uroot -p123456

mysql> grant all privileges on *.*  to ‘temp‘@‘%‘ identified by ‘temp‘ with grant option;
mysql> grant all privileges on *.* to ‘temp‘@‘rac01.com‘ identified by ‘temp‘ with grant option;
mysql> flush privileges;
mysql> use  mysql;
mysql> select host ,user from user;     

進入rac201服務器以下文件夾，執行腳本命令，事先要將mysql-connector-java.jar放入/usr/share/java文件夾中
# cd /opt/cloudera-manager/cm-5.3.6/share/cmf/schema
# ./scm_prepare_database.sh mysql -h rac01.com -utemp -ptemp --scm-host rac01.com scm scm scm

提前下載好CDH源文件
CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
在rac201服務器上創建以下目錄

mkdir -p /opt/cloudera/parcel-repo

# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel* /opt/cloudera/parcel-repo/

在所有服務器上創建
# mkdir -p /opt/cloudera/parcels        

更改parcel目錄的所有人
在rac201服務器上 
# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/
在所有服務器上 
# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels

啟動CM的進程
在rac201服務器上

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d
```
 # ./cloudera-scm-server  start 
```
在所有服務器上啟動agent進程

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d/
```
 # ./cloudera-scm-agent start 
```
CM的服務器登錄界面
http://rac01.com:7180
登錄賬號和密碼 admin admin

技術分享圖片

Cloudera Manager大數據平臺搭建

CM Hadoop 在企業裏快速搭建大數據平臺除了Ambria外還可以用Cloudera Manager，這兩種工具相對於純手工搭建確實便捷很多，但是有利也有弊，相對於新手來說就不便於掌握內部原理，不好排錯等,而純手工搭建，出錯較多，不容易成功。一. 規劃：192.168.3.201

Cloudera Manager大數據平臺搭建

vim /etc/hosts

rpm -qa | grep mysql

wget http://repo.mysql.com/mysql57-community-release-el6-8.noarch.rpm

mkdir -p /opt/cloudera-manager

tar -zxf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz -C /opt/cloudera-manager

vi /opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent/config.ini

Hostname of the CM server.

scp -r /opt/cloudera-manager/ [email protected]:/opt/

useradd --system --home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server --no-create-home --shell=/bin/false cloudera-scm

mysql -uroot -p123456

mkdir -p /opt/cloudera/parcel-repo

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d/

Cloudera Manager大數據平臺搭建

大數據平臺搭建 - cdh5.11.1 - oozie安裝

《Cloudera hadoop大數據平臺實戰指南》此書預計2018年12月底上市

大數據平臺Hadoop的分布式集群環境搭建

【HADOOP】| 環境搭建：從零開始搭建hadoop大數據平臺（單機/偽分布式）-下

大數據平臺學習-1

大數據平臺一鍵安裝OS【定制化OS鏡像制作】

電商大數據平臺運維案例

大數據平臺常用命令

開發人員學Linux(14)：CentOS7安裝配置大數據平臺Hadoop2.9.0

案例分析:大數據平臺技術方案及案例(ppt)

大數據平臺cdh5.3.x 的安裝配置

大數據平臺 CDH5.12.2 安裝部署

大數據平臺粗略架構和數倉設計步驟

CM+CDH大數據平臺

大快大數據平臺架構設計的構成模塊

大數據平臺架構——通用版

大數據平臺技術方案及案例

大快政務大數據平臺解決方案

大數據環境搭建（2）- hive客戶端安裝

Cloudera Manager大數據平臺搭建

vim /etc/hosts

rpm -qa | grep mysql

wget http://repo.mysql.com/mysql57-community-release-el6-8.noarch.rpm

mkdir -p /opt/cloudera-manager

tar -zxf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz -C /opt/cloudera-manager

vi /opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent/config.ini

Hostname of the CM server.

scp -r /opt/cloudera-manager/ [email protected]:/opt/

useradd --system --home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server --no-create-home --shell=/bin/false cloudera-scm

mysql -uroot -p123456

mkdir -p /opt/cloudera/parcel-repo

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d

cd /opt/cloudera-manager/cm-5.3.6/etc/init.d/

相關推薦