系統環境

硬體環境：window下VMware 11虛擬機器

作業系統：紅帽6.5伺服器版本

Cloudera Manager：5.7

CDH：CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel

資料庫：Mysql 5.6

參考說明

下載

Java：http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

MySQL：http://dev.mysql.com/downloads/mysql/

CM5：http://archive.cloudera.com/cm5/

CDH5：http://archive.cloudera.com/cdh5/

JDBC：http://dev.mysql.com/downloads/connector/j/

配置環境

安裝

Cloudera Manager 5

$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/

2. 配置Server地址

$ vi /opt/cm-5.7.0/etc/cloudera-scm-agent/config.ini
server_host=hadoop001
 
server_port=7183

3. MySql Driver

$ tar zxvf mysql-connector-java-5.1.37.tar.gz
$ sudo mkdir -p /usr/share/java/
$ sudo cp mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar /usr/share/java/mysql-connector-java.jar

$ sudo scp -r /opt/* agent_host:/opt/

5. Cloudera Manger和服務需要Linux帳戶來進行管理。所以必須建立一個使用者帳戶在每一個節點機器上，因為Cloudera預設使用這個帳戶來管理Cloudera叢集。

$ sudo useradd --system --home=/opt/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

6. 建立Cloudera Manager Server本地檔案儲存地址

$ sudo mkdir /var/log/cloudera-scm-server
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server

6-1. （可以略過）預設配置使用的路徑

• /var/log/cloudera-scm-headlamp
• /var/log/cloudera-scm-firehose
• /var/log/cloudera-scm-alertpublisher
• /var/log/cloudera-scm-eventserver
• /var/lib/cloudera-scm-headlamp
• /var/lib/cloudera-scm-firehose
• /var/lib/cloudera-scm-alertpublisher
• /var/lib/cloudera-scm-eventserver
• /var/lib/cloudera-scm-server

$ sudo chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp

• Use alternate directories:
  1. If the directories you plan to use do not exist, create them. For example, to create /var/cm_logs/cloudera-scm-headlamp for use by the cloudera-scm user, you can use the following commands:

     mkdir /var/cm_logs/cloudera-scm-headlamp
     chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp

  2. Connect to the Cloudera Manager Admin Console.
  3. Select Clusters > Cloudera Management Service
  4. Select Scope > role name.
  5. Click the Configuration tab.
  6. Enter a term in the Search field to find the settings to be changed. For example, you might enter /var or directory.
  7. Update each value with the new locations for Cloudera Manager to use.Note: The configuration property for the Cloudera Manager Server Local Data Storage Directory (default value is: /var/lib/cloudera-scm-server) is located on a different page:
     1. Select Administration > Settings.
     2. Type directory in the Search box.
     3. Enter the directory path in the Cloudera Manager Server Local Data Storage Directory property.
  8. Click Save Changes to commit the changes.

7. 初始化資料庫及使用者

/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -uroot -p123456 --scm-host hadoop001 root 123456

7-1. （可以略過）初始化資料庫設定 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5

vi /usr/my.cnf

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

7-2. （可以略過）建立自定義資料庫及關聯使用者

CREATE USER 'amon'@'*' IDENTIFIED BY 'amon';
CREATE USER 'rman'@'*' IDENTIFIED BY 'rman';
CREATE USER 'hive'@'*' IDENTIFIED BY 'hive';
CREATE USER 'sentry'@'*' IDENTIFIED BY 'sentry';
CREATE USER 'nav'@'*' IDENTIFIED BY 'nav';
CREATE USER 'navms'@'*' IDENTIFIED BY 'navms';
CREATE USER 'hue'@'*' IDENTIFIED BY 'hue';
CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';

create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';

create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';

create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';

create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';

create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* to 'hue'@'%' identified by 'hue';

create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* to 'oozie'@'%' identified by 'oozie';

FLUSH PRIVILEGES;

$ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
$ cp CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel /opt/cloudera/parcel-repo

9. 啟動主節點Clouder Manager Server

$ /opt/cm-5.7.0/etc/init.d/cloudera-scm-server start

10. 啟動每個節點的Clouder Manager Agent

$ mkdir /opt/cm-5.7.0/run/cloudera-scm-agent
$ /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start

設定自動啟動Agent

$ cp /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
$ chkconfig cloudera-scm-agent on

Cloudera Distribution Hadoop 5.7.0

1. 登入localhost:7180

2. 有3個版本可以使用就不解釋了

3. 檢視到可以用到的元件

4. 選擇需要安裝的節點

5. 安裝CDH 5.7.0並根據需求安裝相應元件

6. 分配節點任務