1. 程式人生 > >最詳細的Hadoop+Hbase+Hive完全分散式環境搭建教程(二)

最詳細的Hadoop+Hbase+Hive完全分散式環境搭建教程(二)

繼續上次的教程,這次安裝hbase與hive

一、hbase2.1.1安裝

在master節點:

$ tar -zxvf hbase-2.1.1-bin.tar.gz -C /opt
$ cd /opt/hbase-2.1.1/conf/
$ vi hbase-env.sh 

將下面這句話加到檔案末尾 export JAVA_HOME=/opt/jdk1.8.0_102

$ vi hbase-site.xml

修改如下:

<configuration>
<property>
  <name>hbase.rootdir</name>
  <value>hdfs://master:9000/hbase</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>master,slave1,slave2</value>
</property>
<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/home/fay/zookeeper</value>
</property>
</configuration>

$ vi regionservers 修改如下:

# 刪掉localhost
slave1
slave2

$ vi backup-masters 這個檔案是新建的,加入下面這句話

slave1

然後就是配置環境變數: $ vi ~/.bashrc

export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:

$ source ~/.bashrc 然後將配置檔案和整個目錄給其它兩個節點:

$ scp -r hbase-2.1.1/ [email protected]:/opt
$ scp -r hbase-2.1.1/ [email protected]:/opt
$ scp ~/.bashrc [email protected]:/home/fay
$ scp ~/.bashrc [email protected]:/home/fay

另外兩個節點source一下 $ source ~/.bashrc 然後在hadoop起來的情況下,再啟動hbase

# 啟動hadoop
$ start-dfs.sh
$ start-yarn.sh
# 啟動hbase
$ start-hbase.sh

然後master節點 :

[[email protected] opt]$ jps
39825 Jps
38898 ResourceManager
39398 HQuorumPeer #hbase的zookeeper例項
38487 NameNode
39495 HMaster # hbase master
38702 SecondaryNameNode

slave節點:

[[email protected] ~]$ jps
16481 DataNode
16598 NodeManager
16886 HRegionServer # hbase regionserver
16775 HQuorumPeer 
17003 HMaster # 這個是slave1,因為前面配置了backup-masters,slave2沒有這個
17278 Jps

這裡注意時間同步問題,啟動可能比較耗時,不要輕易ctrl+c取消,可能會異常。好了,測試一下

[[email protected] opt]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase-2.1.1/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.1, rb60a92d6864ef27295027f5961cb46f9162d7637, Fri Oct 26 19:27:03 PDT 2018
Took 0.0047 seconds                                                                                                                                               
hbase(main):001:0> list
TABLE                                                                                                                                                             
wuxia                                                                                                                                                             
1 row(s) # 這裡我之前建立了表,初始狀態應該是0
Took 1.8662 seconds                                                                                                                                               
=> ["wuxia"]
hbase(main):002:0> create 'hello','world'
Created table hello
Took 3.1974 seconds                                                                                                                                               
=> Hbase::Table - hello

不出異常基本上hbase 搭好了

二、hive2.3.4搭建

1. 安裝mysql 8.0.13 (不安裝可以使用自帶的derby)

解除安裝系統自帶的mariadb-lib 檢視mariadb版本 rpm -qa|grep mariadb

解除安裝mariadb rpm -e mariadb-libs-5.5.56-2.el7.x86_64 --nodeps mysql有依賴關係,安裝順序是 common、lib、client、server

sudo  rpm -ivh mysql-community-common-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-client-8.0.13-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-server-8.0.13-1.el7.x86_64.rpm

啟動mysql:

sudo systemctl start mysqld 設定root密碼:

 [[email protected] ~]$ cat /var/log/mysqld.log
2018-11-11T09:12:06.253251Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.13) initializing of server in progress as process 20113
2018-11-11T09:12:09.192846Z 5 [Note] [MY-010454] [Server] A temporary password is generated for [email protected]: pR+ndHtvO6#d

上面的pR+ndHtvO6#d就是root初始密碼,用這個密碼登入mysql

[[email protected] ~]$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 8.0.13 MySQL Community Server - GPL

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> 

然後就是建立一個fay使用者,並給予他許可權

mysql> CREATE USER 'fay'@'master' IDENTIFIED BY '你的密碼';
mysql> create database hive_metedata
mysql> grant all privileges on *.* to 'fay'@'master';
mysql> flush privileges;

2. 安裝hive

解壓到/opt資料夾: $ tar -zxvf apache-hive-2.3.4-bin.tar.gz -C /opt $ cd /opt/apache-hive-2.3.4-bin/conf/ 修改 hive-env.sh 配置hadoop_home等環境變數,在檔案末尾有

[[email protected] conf]$ cp hive-env.sh.template hive-env.sh
[[email protected] conf]$ vi hive-env.sh
...
# HADOOP_HOME=${bin}/../../hadoop
export HADOOP_HOME=/opt/hadoop-2.8.5

# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/apache-hive-2.3.4-bin/conf

# Folder containing extra libraries required for hive compilation/execution can be controlled by:
export HIVE_AUX_JARS_PATH=/opt/apache-hive-2.3.4-bin/lib
export JAVA_HOME=/opt/jdk1.8.0_102

修改hive-site.xml

[[email protected] conf]$ cp hive-default.xml.template hive-site.xml 
[[email protected] conf]$ vi hive-site.xml 
# 檔案太長了,就copy幾個重要需要改的地方,找到這些配置項改
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/fay/hive/tmp/fay</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/fay/hive/tmp/${hive.session.id}_resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
</property>
 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>你的密碼</value>
    <description>password to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hive_metedata?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>    
    <description>Driver class name for a JDBC metastore</description>
  </property>

:javax.jdo.option.ConnectionDriverName如果是com.mysql.jdbc.Driver, 就要把那個mysql-connector jar包放入到${hive_home}/lib資料夾裡,所以那個mysql-connector jar包可有可無

然後將配置檔案中所有的${system:java.io.tmpdir}更改為 /home/fay/hive/tmp (如果沒有該檔案則建立),並將此資料夾賦予讀寫許可權,將${system:user.name}更改為 fay

然後新增環境變數:

[[email protected] lib]$ vi ~/.bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions

export JAVA_HOME=/opt/jdk1.8.0_102
export HADOOP_HOME=/opt/hadoop-2.8.5
export HBASE_HOME=/opt/hbase-2.1.1
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HIVE_HOME=/opt/apache-hive-2.3.4-bin
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin

source ~/.bashrc 建立hdfs資料夾,並賦予許可權

$ hadoop fs -mkdir -p /user/hive/
$ hadoop fs -mkdir -p /user/hive/warehouse
$ hadoop fs -chmod 777 /user/hive/
$ hadoop fs -chmod 777 /user/hive/warehouse

初始化hive,然後啟動hive:

[[email protected] conf]$ schematool -dbType mysql -initSchema
[[email protected] conf]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.4-bin/lib/hive-common-2.3.4.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> 

然後測試下hive的功能,與sql差不多,這裡不過多介紹,到這裡,全部環境搭建完啦。如果中途遇到問題,網上還是有不少解決方案的,後面如果我按這個再搭一次環境出問題再來調整。環境多搭建就熟悉了,我也是搭建過將近10次的環境,以前都是用一個版本,這次打算用最新的版本,中間還是有不少坑的。