本博文的主要內容是：

　　.複習HA相關

　　.MySQL資料庫

　　.先在MySQL資料庫中建立hive資料庫

　　.hive的配置

以下是Apache Hadoop HA的總結。分為hdfs HA和yarn HA。

 

 　　以上，是參考《Hadoop海量資料處理  技術詳解與專案實戰》

　　強烈建議，先看

　　想說的是，hive只是個工具，包括它的資料分析，依賴於mapreduce，它的資料管理，依賴於外部系統。

　　metastore_db，是在哪目錄下執行，在哪裡產生資料。

 

　　由此可見，你在哪路徑下，執行hive指令，就在哪路徑下生成metastore_db。建一套資料庫檔案，這樣是極其不合適的，公司裡每個人若不一樣，則會顯得非常混雜。導致員工之間無法公用交流。

　　為此，需公用的，mysql。作為hive的元資料管理。

       若什麼都不做，則hive預設是用derby，是單使用者，不方便，不適合多使用者。

　　說明的是，關於hive的安裝和mysql的安裝，一般，都是先安裝好hive，再來安裝mysql。當然，我也看到他人是反著的也可以。

配置mysql metastore（切換到root使用者）  

         配置HIVE_HOME環境變數

1、線上安裝Mysql資料庫

　　如果你是一名有經驗大資料工程師，無論是安裝jdk（注意：CentOS6.5有自帶的jdk），還是安裝mysql。都是先檢視系統是否已經安裝Mysql包。

[[email protected] app]# rpm -qa|grep mysql
mysql-libs-5.1.71-1.el6.x86_64
[[email protected] app]# rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
[[email protected] app]# rpm -qa|grep mysql

[[email protected] app]#

[[email protected] app]# pwd

/home/hadoop/app

[[email protected] app]# yum -y install mysql-server  （CentOS版本）

　　若是Ubuntu系統的話，見

或者，可以這麼來安裝。

但是這兩個得，提前下載好。

rpm -ivh MySQL-server-5.1.73-1.glibc23.i386.rpm 
rpm -ivh MySQL-client-5.1.73-1.glibc23.i386.rpm

 參考：http://blog.csdn.net/u014726937/article/details/52142048

 具體，見

2、啟動MySQL服務

 

[[email protected] app]# service mysqld start （CentOS版本）

Initializing MySQL database:  Installing MySQL system tables...

OK

Filling help tables...

OK

To start mysqld at boot time you have to copy

support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !

To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'

/usr/bin/mysqladmin -u root -h weekend110 password 'new-password'

Alternatively you can run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test

databases and anonymous user created by default.  This is

strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:

cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl

cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

[  OK  ]

Starting mysqld:  [  OK  ]

[[email protected] app]#

[[email protected] app]# mysql -u root -p        進入

Enter password: （回車）

 其實啊，這裡啊，直接幾步就可以了

mysql> CREATE USER  'hive'@'%' IDENTIFIED BY 'hive';    //建立一個賬號：使用者名稱為hive，密碼為hive

mysql> GRANT ALL PRIVILEGES ON  *.* to  'hive'@'%' IDENTIFIED BY  'hive' WITH GRANT OPTION;   //將許可權授予host為'weekend110'的hive使用者

mysql> FLUSH PRIVILEGES;

mysql> exit;

 其中， *.*:所有庫下的所有表       %：任何IP地址或主機都可以連線

[[email protected] app]# mysql -u root -p
Enter password:   （回車）
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON *.* to 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> exit;
Bye
[[email protected] app]#

 　　此外，為了使遠端使用者可以訪問MySQL，需要修改 /etc/mysql/my.cnf（Ubuntu系統）或/etc/my.cnf（CentOS系統），將 bind-address一行註釋掉。

當然，這可以以後更改，也可以的。

　　可以看出，CentOS系統裡，是沒有的。

 

 　　sudo /etc/init.d/mysql restart  進行重啟、

3、設定MySQL的root使用者設定密碼

　　MySQL在剛剛被安裝的時候，它的 root 使用者是沒有被設定密碼的（預設密碼為空，是回車）。首先來設定 MySQL的root 使用者的root密碼。

[[email protected] app]# mysql -u root -p

Enter password: （回車）

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 2

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select user,host,password from mysql.user;

+------+------------+----------+

| user | host       | password |

+------+------------+----------+

| root | localhost  |          |

| root | weekend110 |          |

| root | 127.0.0.1  |          |

|      | localhost  |          |

|      | weekend110 |          |

+------+------------+----------+

5 rows in set (0.00 sec)

mysql> set password for [email protected]=password('rootroot');

Query OK, 0 rows affected (0.00 sec)

mysql> set password for [email protected]=password('rootroot');

Query OK, 0 rows affected (0.00 sec)

mysql> select user,host,password from mysql.user;

+------+------------+-------------------------------------------+

| user | host       | password                                  |

+------+------------+-------------------------------------------+

| root | localhost  | *6C362347EBEAA7DF44F6D34884615A35095E80EB |

| root | weekend110 | *6C362347EBEAA7DF44F6D34884615A35095E80EB |

| root | 127.0.0.1  |                                           |

|      | localhost  |                                           |

|      | weekend110 |                                           |

+------+------------+-------------------------------------------+

5 rows in set (0.00 sec)

mysql>

4、為 Hive 建立相應的 Mysql 賬戶hive，再設定密碼hive。

 

mysql> create user 'hive' identified by 'hive';    //建立一個賬號：使用者名稱為hive，密碼為hive

Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* to 'hive'@'weekend110' identified by 'hive';  //將許可權授予host為'weekend110'的hive使用者

Query OK, 0 rows affected (0.00 sec)

mysql> select user,host,password from mysql.user;

+------+------------+-------------------------------------------+

| user | host       | password                                  |

+------+------------+-------------------------------------------+

| root | localhost  | *6C362347EBEAA7DF44F6D34884615A35095E80EB |

| root | weekend110 | *6C362347EBEAA7DF44F6D34884615A35095E80EB |

| root | 127.0.0.1  |                                           |

|      | localhost  |                                           |

|      | weekend110 |                                           |

| hive | %          | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |

| hive | weekend110 | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |

+------+------------+-------------------------------------------+

7 rows in set (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.01 sec)

mysql> exit;

Bye

[[email protected] app]#

5、用剛才建立的 “hive” 賬號登入，建立 Hive 專用的元資料庫

 

[[email protected] app]# mysql -uhive -hweekend110 -phive

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 3

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE hive;     //建立 hive使用者，專用的元資料庫hive

Query OK, 1 row affected (0.01 sec)

mysql> SHOW DATABASES;   （規範，大寫）

+--------------------+

| Database           |

+--------------------+

| information_schema |

| hive               |

| mysql              |

| test               |

+--------------------+

4 rows in set (0.03 sec)

　　其中，預設建立好的是3個，information_schema、mysql、test。這三個資料庫是MySQL安裝程式自動建立的。

其中，mysql庫中包含的是5個MySQL授權表，information_schema庫中是相關資訊，而test是供使用者練習使用的。

mysql> exit;

Bye

[[email protected] app]#

[[email protected] app]# mysql -u hive -h weekend110 -p hive

Enter password:

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 6

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit;

Bye

[[email protected] app]# mysql -uhive  -phive

ERROR 1045 (28000): Access denied for user 'hive'@'localhost' (using password: YES)

[[email protected] app]# mysql -uroot  -prootroot

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 8

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all on *.* to 'hive'@'localhost' identified by 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> set password for [email protected]=password('hive');
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit;

Bye

[[email protected] app]#

最後結束，情況圖是如下（此博文完全正確配法）

　　由此，可見啊，

　　如果有誤，請見部落格

配置hive

學個技巧，如何快速的搜尋。

　　先按Esc，再按Shift，再 . 鍵 +  / 鍵

更改四個地方： （新手，只需配置這四個地方，即可）

javax.jdo.option.ConnectionURL

javax.jdo.option.ConnectionDriverName

javax.jdo.option.ConnectionUserName

javax.jdo.option.ConnectionPassword

 　　當然，有些人說，要改5個地方，甚至，更多地方。

<property>

  <name>javax.jdo.option.ConnectionURL</name>

  <value>jdbc:derby:;databaseName=metastore_db;create=true</value>

  <description>JDBC connect string for a JDBC metastore</description>

</property>

改為

<property>

  <name>javax.jdo.option.ConnectionURL</name>

  <value>jdbc:mysql://weekend110:3306/hive?createDatabaseIfNotExist=true</value>

  <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>org.apache.derby.jdbc.EmbeddedDriver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

改為

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>com.mysql.jdbc.Driver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>APP</value>

  <description>username to use against metastore database</description>

</property>

改為

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>hive</value>

  <description>username to use against metastore database</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>mine</value>

  <description>password to use against metastore database</description>

</property>

改為

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>hive</value>

  <description>password to use against metastore database</description>

</property>

 

之後，會出現如下問題。

 

還需驅動mysql的jar包，

自帶的是derby的

 

 　　也許，最好執行時，會出現如下問題。

　步驟二： 　則，需，建立一個目錄，如我這裡，是/home/spark/app/hive-1.2.1/iotmp

< property>

         < name>hive.querylog.location< /name>

         < value>/home/spark/app/hive-1.2.1/iotmp< /value>

         < description>Location of Hive run time structured log file< /description>

< /property>

< property>

         < name>hive.exec.local.scratchdir< /name>

         < value>/home/spark/app/hive-1.2.1/iotmp< /value>

         < description>Local scratch space for Hive jobs< /description>

< /property>

< property>

         < name>hive.downloaded.resources.dir< /name>

         < value>/home/spark/app/hive-1.2.1/iotmp< /value>

         < description>Temporary local directory for added resources in the remote file system.< /description>

< /property>

 

　步驟四：解決方法：jline版本不一致，把HADOOP_HOME/share/hadoop/yarn/lib和HIVE_HOME/lib下的jline-**.jar版本一致就可以了，複製其中一個高版本覆蓋另一個。

　　即，取其中的高版本即可。我這裡的情況是，HADOOP_HOME/share/hadoop/yarn/lib下是0.9多，刪除這個。

　　將HIVE_HOME/lib下的jline-2.12.jar，複製一份到HADOOP_HOME/share/hadoop/yarn/lib下。

root使用者來，驗證一下

[[email protected] bin]# pwd

/home/hadoop/app/hive-0.12.0/bin

[[email protected] bin]# ./hive

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

16/10/10 09:26:36 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Logging initialized using configuration in jar:file:/home/hadoop/app/hive-0.12.0/lib/hive-common-0.12.0.jar!/hive-log4j.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/hadoop/app/hive-0.12.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hive> exit;

[[email protected] bin]#

hadoop使用者，來驗證一下

[[email protected] bin]$ pwd

/home/hadoop/app/hive-0.12.0/bin

[[email protected] bin]$ ./hive

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

16/10/10 09:28:16 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

Logging initialized using configuration in jar:file:/home/hadoop/app/hive-0.12.0/lib/hive-common-0.12.0.jar!/hive-log4j.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/hadoop/app/hive-0.12.0/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hive> exit;

[[email protected] bin]$

　　　總結，想說的是，在單節點叢集裡。hadoop和hive都是在一個節點裡。

如上配置，或

<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://weekend110:3306/hive?useUnicode=true&amp;characterEncoding=UTF-8&amp;createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>

　　直接啟動沒問題。

但是，我在3節點（HadoopMaster、HadoopSlave1、HadoopSlave2）上，的HadoopSlave1上這麼配置，再啟動。

一直報錯誤！

　　也許是沒在主節點上配置吧！