Hive本地模式安裝配置
阿新 • • 發佈：2019-02-18
準備：
基本步驟：
一. 解除安裝mysql
二. 安裝mysql
三. 安裝hive
一. 解除安裝mysql
#1.檢查是否已經安裝MySQL元件 [root@sst_search_a /]# rpm -qa | grep -i mysql MySQL-devel-5.6.23-1.linux_glibc2.5 MySQL-client-5.6.23-1.linux_glibc2.5 MySQL-server-5.6.23-1.linux_glibc2.5 #2.解除安裝前關閉MySQL服務 [root@sst_search_a /]# service mysql status MySQL running (25673)[ OK ] [root@sst_search_a /]# service mysql stop Shutting down MySQL..[ OK ] [root@sst_search_a /]# service mysql status MySQL is not running[FAILED] #3.收集MySQL對應資料夾資訊 [root@sst_search_a /]# find / -name mysql /etc/rc.d/init.d/mysql /etc/logrotate.d/mysql /var/lock/subsys/mysql /var/lib/mysql /var/lib/mysql/mysql /usr/include/mysql /usr/include/mysql/mysql /usr/bin/mysql /usr/share/mysql /usr/lib64/mysql #4.解除安裝MySQL各類元件及資料夾 #刪除元件 [root@sst_search_a /]# rpm -ev MySQL-devel-5.6.23-1.linux_glibc2.5 [root@sst_search_a /]# rpm -ev MySQL-server-5.6.23-1.linux_glibc2.5 [root@sst_search_a /]# rpm -ev MySQL-client-5.6.23-1.linux_glibc2.5 #刪除資料夾 [root@DB-Server init.d]# find / -name mysql /var/lib/mysql /var/lib/mysql/mysql /usr/lib64/mysql [root@DB-Server init.d]# rm -rf /var/lib/mysql [root@DB-Server init.d]# rm -rf /var/lib/mysql/mysql [root@DB-Server init.d]# rm -rf /usr/lib64/mysql #5.確認MySQL徹底刪除 [root@sst_search_a /]# rpm -qa | grep -i mysql
二、 安裝MySQL
三、安裝Hive#當前為root賬戶 #1.安裝server [root@sst_search_a hive]# rpm -ivh /opt/MySQL-server-5.5.15-1.linux2.6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-server ########################################### [100%] ...... /usr/bin/mysqladmin -u root password 'new-password' /usr/bin/mysqladmin -u root -h test1 password 'new-password' ...... #檢視mysql3306埠 [root@sst_search_a hive]# netstat -nat tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN #2.安裝client [root@sst_search_a hive]# rpm -ivh /opt/ MySQL-client-5.5.15-1.linux2.6.x86_64.rpm warning: MySQL-client-5.1.7-0.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5 Preparing... ########################################### [100%] 1:MySQL-client ########################################### [100%] #3.修改密碼 [root@sst_search_a hive]# usr/bin/mysqladmin -u root password 'new-password' #mysqladmin -u使用者名稱 -p舊密碼 password 新密碼 [root@sst_search_a hive]# mysql -uroot -new-password mysql> insert into mysql.user(Host,User,Password) values("localhost","hive",password("hive")); mysql> create database hive; mysql> grant all on hive.* to hive@'%' identified by 'hive'; mysql> grant all on hive.* to hive@'localhost' identified by 'hive'; mysql> flush privileges; mysql> exit; [root@sst_search_a hive]# mysql -uhive -phive mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | hive | | test | +--------------------+ 3 rows in set (0.00 sec) mysql> exit;
#切換至hadoop賬戶 [hadoop@sst_search_a ~]$ vi .bash_profile #HIVE_HOME export HIVE_HOME=/opt/hive export PATH=$PATH:$HIVE_HOME/bin [hadoop@sst_search_a ~]$ source .bash_profile #配置hive-env.sh [hadoop@sst_search_a conf]$ cp hive-env.sh.template hive-env.sh [hadoop@sst_search_a conf]$ vi hive-env.sh export HIVE_HOME=/opt/hive [hadoop@sst_search_a conf]$ source hive-env.sh #配置hive-site.xml [hadoop@sst_search_a conf]$ cp hive-default.xml.template hive-site.xml [hadoop@sst_search_a conf]$ vi hive-site.xml #檔案比較大，使用'/javax.jdo.option.ConnectionURL'查詢 <property> <name>javax.jdo.option.ConnectionURL </name> <value>jdbc:mysql://localhost:3306/hive </value> </property> <property> <name>javax.jdo.option.ConnectionDriverName </name> <value>com.mysql.jdbc.Driver </value> </property> <property> <name>javax.jdo.option.ConnectionPassword </name> <value>hive </value> </property> <property> <name>hive.hwi.listen.port </name> <value>9999 </value> <description>This is the port the Hive Web Interface will listen on </descript ion> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>true</value> </property> <property> <name>datanucleus.fixedDatastore </name> <value>false</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>Username to use against metastore database</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/home/hdpsrc/hive/iotmp</value> <description>Location of Hive run time structured log file</description> </property> #拷貝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面 [hadoop@sst_search_a conf]$ mv /opt/Desktop/mysql-connector-java-5.1.6-bin.jar /opt/hive/lib/ #把jline-2.12.jar拷貝到hadoop相應的目錄下，替代jline-0.9.94.jar，否則啟動會報錯 [hadoop@sst_search_a conf]$ cp /opt/hive/lib/jline-2.12.jar /opt/hadoop-2.6.0/share/hadoop/yarn/lib/ [hadoop@sst_search_a conf]$ mv /opt/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /opt/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak / #建立hive臨時資料夾 [hadoop@sst_search_a conf]$ mkdir /opt/hive/iotmp #啟動hadoop後測試hive [hadoop@sst_search_a conf]$ hive hive> show databases; OK default Time taken: 0.907 seconds, Fetched: 1 row(s)
四、遇到的問題
問題1：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/........../lib/slf4j-log4j12-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/........../slf4j-log4j12-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
解決辦法：
jar包重複，刪掉了hive/lib資料夾下的對應jar包
問題2：
Logging initialized using configuration in jar:file:/opt/tuniu/hive/apache-hive-2.1.0-bin/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Permission denied
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at org.apache.hadoop.hive.common.FileUtils.createTempFile(FileUtils.java:785)
at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:904)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:555)
... 9 more
解決辦法：
注意到permission denied,一定是許可權問題。猜測可能是hadoop使用者對hive資料夾許可權不夠，暴力賦予了足夠的許可權
chmod 777 -R /opt/hive
（注意這裡其實問題變形很多，還是得讀一下自己的錯誤日誌）
問題3：
which: no hbase in (/opt/tuniu/java/jdk1.8.0_101/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/tuniu/hadoop/hadoop-2.6.4/bin:/opt/tuniu/hadoop/hadoop-2.6.4/sbin:/root/bin:/opt/tuniu/hive/apache-hive-2.1.0-bin/bin)
還沒搞明白和hbase有什麼關係。。。不影響使用，暫時擱置。