一、hive相關知識複習

複習一下hadoop的資料倉庫hive，它的元資料儲存在第三方資料庫中，實際資料儲存在hadoop的hdfs上，所以hive不能脫離hadoop與第三方資料庫單獨存在，我們前面已經安裝好了hadoop2.7.3版本，只要安裝一個第三方資料庫即可滿足hive安裝的依賴環境，這裡我選擇安裝的是mysql5.5.46，hive選擇安裝的是1.2.1版本。

hive將結構化的資料檔案對映為資料庫表（元資料儲存在mysql中），資料檔案在hdfs上儲存，通過將sql語句解析轉換為mapreduce任務進行分散式計算執行。

hive官網對安裝準備有如下描述;

Hive installation has these requirements:

• Java 1.7 (preferred).
  Note:  Hive versions 1.2 onward require Java 1.7 or newer. Hive versions 0.14 to 1.1 work with Java 1.6, but prefer 1.7. Users are strongly advised to start moving to Java 1.8 (see HIVE-8607). 
• Hadoop 2.x (preferred), 1.x (not supported by Hive 2.0.0 onward).
  Hive versions up to 0.13 also supported Hadoop 0.20.x, 0.23.x.
• Hive is commonly used in production Linux and Windows environment. Mac is a commonly used development environment. The instructions in this document are applicable to Linux and Mac. Using it on Windows would require slightly different steps.  

二、安裝檔案下載連結

hive：

http://apache.fayea.com/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz

mysql：

https://downloads.mysql.com/archives/get/file/MySQL-server-5.5.46-1.el7.x86_64.rpm

https://downloads.mysql.com/archives/get/file/MySQL-client-5.5.46-1.el7.x86_64.rpm

三、安裝步驟

1、解除安裝作業系統中預安裝的mysql依賴庫

--下面操作均需要在root許可權下操作

#yum remove mysql-libs 

2、安裝mysql客戶端

[[email protected] hadoop]#  rpm -ivh MySQL-client-5.5.46-1.el7.x86_64.rpm 

Preparing...                          ################################# [100%]
Updating / installing...
1:MySQL-client-5.5.46-1.el7        ################################# [100%]

--slave01上也安裝mysql的客戶端

[[email protected] hadoop]#  rpm -ivh MySQL-client-5.5.46-1.el7.x86_64.rpm 

Preparing...                          ################################# [100%]
Updating / installing...
1:MySQL-client-5.5.46-1.el7        ################################# [100%]

3、安裝mysql伺服器端

[[email protected] hadoop]# rpm -ivh MySQL-server-5.5.46-1.el7.x86_64.rpm 
warning: MySQL-server-5.5.46-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:MySQL-server-5.5.46-1.el7        ################################# [100%]
170404  1:49:51 [Note] /usr/sbin/mysqld (mysqld 5.5.46) starting as process 33406 ...
170404  1:49:51 [Note] /usr/sbin/mysqld (mysqld 5.5.46) starting as process 33413 ...

--mysql安裝成功後輸出資訊，提示要首先啟動服務為mysql的root使用者設定密碼
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

--mysql怕我不知道怎麼為root設定密碼，還非常慷慨的給出了例項
/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h master password 'new-password'

--如果上面給出的更改密碼方法你不爽的話，mysql還提供了套餐服務
Alternatively you can run:
/usr/bin/mysql_secure_installation
--下面還非常細心的建議在生產資料庫中刪除test庫等必要的安全措施
which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.


See the manual for more instructions.
Please report any problems at http://bugs.mysql.com/

4、啟動mysql資料庫服務

[[email protected] hadoop]# service mysql start

Starting MySQL... SUCCESS! 

5、設定root使用者的登陸密碼,這一步開始不需要在root許可權下運行了

[[email protected] ~]$ /usr/bin/mysqladmin -u root password 'mysql123'

--我的密碼是為我這個老年痴呆症患者設計的

6、用root使用者測試進入mysql資料庫

[[email protected] ~]$ mysql -uroot -pmysql123
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.46 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.02 sec)

7、建立hive元資料儲存資料庫，並授權hive使用者訪問

--建立資料庫hivemeta

mysql> create database hivemeta;
Query OK, 1 row affected (0.05 sec)

--老版本的mysql安裝成hive的元資料儲存必須要設定資料庫字符集為latin1，否則在hive中使用時會報錯

mysql> alter database hivemeta character set latin1;
Query OK, 1 row affected (0.00 sec)

--建立mysql資料庫登陸hive使用者

mysql> create user 'hive'@'%' identified by 'mysql123';

Query OK, 0 rows affected (0.05 sec)

Query OK, 0 rows affected (0.05 sec)

--下面的重點讓我跳了2晚上的坑，就是因為對mysql的使用者登陸授權機制搞清晰；

grant all privileges on *.* to 'hive'@'localhost' identified by 'mysql123' with grant option;

grant all privileges on *.* to 'hive'@'master' identified by 'mysql123' with grant option;

正常情況下使用%授權應該指的是對所有主機授權，但是查詢mysql的官方文件說明，%並不包括localhost和其他的域名，如果使用域名登陸必須要對域名進行單獨授權；

%實際值得應該是IP地址。

驗證測試：

mysql -h localhost -u hive -p

mysql -h master -u hive -p

如果不加上上面的授權，mysql登陸就會自己報錯，hive其實是通過jdbc呼叫，如果使用mysql命令列都會報錯，hive肯定也會呼叫失敗。

--重新整理許可權到資料庫中，設定生效

mysql> flush privileges;
Query OK, 0 rows affected (0.03 sec)

--驗證hive使用者授權情況，應該出現允許%，localhost，master三條記錄

mysql> select user,host from mysql.user;
+------+-----------+
| user | host      |
+------+-----------+
| hive | %         |
| root | 127.0.0.1 |
| root | ::1       |
|      | localhost |
| hive | localhost |
| root | localhost |
|      | master    |
| hive | master    |
| root | master    |
+------+-----------+
9 rows in set (0.10 sec)

8、配置叢集hive的作業系統環境變數

vim /home/hadoop/.bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_121
export HADOOP_HOME=/home/hadoop/bigdata/hadoop
export HADOOP_USER_NAME=hadoop
export HIVE_HOME=/home/hadoop/bigdata/hive
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$PATH

--匯入環境變數檔案

source /home/hadoop/.bashrc

--將環境配置檔案拷貝到slave01、slave02的對應目錄下

[[email protected] ~]$ scp .bashrc [email protected]:/home/hadoop/
.bashrc                                       100%  477     0.5KB/s   00:00    
[[email protected] ~]$ scp .bashrc [email protected]:/home/hadoop/
.bashrc                                       100%  477     0.5KB/s   00:00 

9、解壓縮hive的安裝檔案，開始hive的解壓安裝

cd /home/hadoop/bigdata/

tar -zxf apache-hive-1.2.1-bin.tar.gz

mv apache-hive-1.2.1-bin hive

cd /home/hadoop/bigdata/hive/conf

--根據配置檔案模板建立hive執行配置檔案方便下一步填充必要內容

cp hive-default.xml.template hive-site.xml

cp hive-env.sh.template hive-env.sh

cp hive-log4j.properties.template hive-log4j.properties

10、複製mysql的jdbc驅動JAR包檔案到/home/hadoop/bigdata/hive/lib目錄下

[[email protected] lib]$ ls -la mysql-connector-java-5.1.7-bin.jar 
-rw-rw-r-- 1 hadoop hadoop 709922 Apr  5 06:12 mysql-connector-java-5.1.7-bin.jar

--因為mysql安裝時預設不帶這個jar包檔案，所以需要單獨從oracle的網站上去下載，沒有這個JAR包hive無法正常呼叫mysql根本無法啟動。

11、修改hive的配置檔案

--配置hive執行環境

vim /home/hadoop/bigdata/hive/conf/hive-env.sh

export HADOOP_HOME=/home/hadoop/bigdata/hadoop
export HIVE_CONF_DIR=/home/hadoop/bigdata/hive/conf

--配置hive日誌log4j框架的日誌儲存路徑

vim /home/hadoop/bigdata/hive/conf/hive-log4j.properties

hive.log.threshold=ALL
hive.root.logger=INFO,DRFA
hive.log.dir=/home/hadoop/bigdata/hive/log
hive.log.file=hive.log

--配置hive執行主配置檔案

vim /home/hadoop/bigdata/hive/conf/hive-site.xml

 <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>hdfs://master:9000/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://master:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>

--Hive使用此目錄來儲存用於查詢的不同map/reduce階段的計劃以及儲存這些階段的中間輸出，這裡是hdfs的目的地。

  <property>
    <name>hive.exec.scratchdir</name>
    <value>hdfs://master:9000/hive/scratchdir</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
  </property>

  <property>
    <name>hive.querylog.location</name>
    <value>/home/hadoop/bigdata/hive/logs</value>
    <description>Location of Hive run time structured log file</description>
  </property>

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://master:3306/hivemeta?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>

 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>mysql123</value>
    <description>password to use against metastore database</description>
  </property>

--當Hive以本地模式執行時，此目錄用於臨時檔案

<property>

    <name>hive.exec.local.scratchdir</name>

    <value>/home/hadoop/bigdata/hive/logs</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/hadoop/bigdata/hive/logs</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>

12、將hive安裝檔案拷貝到slave01上，使得slave01可以使用hive

scp -r /home/hadoop/bigdata/hive [email protected]:/home/hadoop/bigdata/

13、在master上啟動metastore服務

[[email protected] ~]$ hive --service metastore
Starting Hive Metastore Server

--另起一個終端視窗驗證hive的metastore服務是否啟動成功

[[email protected] lib]$ ps -ef | grep hive
hadoop    29327  27579  0 04:17 pts/1    00:00:24 /usr/java/jdk1.8.0_121/bin/java -Xmx256m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/bigdata/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/bigdata/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/bigdata/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /home/hadoop/bigdata/hive/lib/hive-service-1.2.1.jar org.apache.hadoop.hive.metastore.HiveMetaStore

--如果驗證正常就可以使用下面的方法在後臺啟動服務

nohup hive -service metastore&

14、在slave01上啟動Hiveserver2服務

[[email protected] ~]$ hive --service hiveserver2

--另起一個終端視窗驗證hive的hiveserver2服務是否啟動成功

[[email protected] bin]$ ps -ef | grep hive
hadoop    18552  16412  2 07:08 pts/2    00:00:07 /usr/java/jdk1.8.0_121/bin/java -Xmx256m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/bigdata/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/bigdata/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/bigdata/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /home/hadoop/bigdata/hive/lib/hive-service-1.2.1.jar org.apache.hive.service.server.HiveServer2

--同樣如果啟動正常可以使用下面的後臺啟動方法

nohup hive  --service hiveserver2&

15、在master或slave01上進入hive客戶端驗證

hive

hive> create database test;
OK
Time taken: 0.392 seconds
hive> show databases;
OK
default
test
Time taken: 0.023 seconds, Fetched: 2 row(s)

三、故障排錯

--在master主機上使用master域名無法登陸，還是mysql授權問題導致

[[email protected] ~]$ mysql -h master -u hive -pmysql123
ERROR 1045 (28000): Access denied for user 'hive'@'master' (using password: YES)

--mysql對使用者授權出錯導致hive啟動報錯，解決方案參見第7步

Exception in thread "main" java.lang.RuntimeException: java.net.ConnectException: Call From slave01/192.168.10.167 to master:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Call From slave01/192.168.10.167 to master:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

--沒有配置hive-site.xml檔案中的hive.exec.local.scratchdir與hive.downloaded.resources.dir導致hive啟動報錯，參見第11步；

Logging initialized using configuration in file:/home/hadoop/bigdata/hive/conf/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:563)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 8 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)
... 11 more

四、總結

       hive的安裝與第三方的環境結合太過緊密，尤其是mysql，如果對mysql的安裝授權不太清楚會走很多彎路，首先在master節點上解壓安裝配置好hive，並且不要忘記下載mysql的jdbc驅動檔案並把檔案放置在hive的lib目錄下，不要著急啟動hive的服務，首先檢查mysql的使用者授權是否完整，並在客戶端下測試使用主機名是否可以正常登陸，登陸後是否可以開啟hivemeta資料庫，這一步測試完成後再開始在master節點上啟動hive的metastore服務，一開始不要使用後臺啟動方法，如果出錯可以直接看到丟擲的異常方便查詢故障，所有的安裝除錯過程需要對每一步的原理與細節弄得非常清晰否則到處都是坑。