HIVE與mysql的關係 hive常用命令整理 hive與hdfs整合過程

阿新 • • 發佈：2019-02-16

轉：https://my.oschina.net/winHerson/blog/190131

二、hive常用命令
1. 開啟行轉列功能之後:
set hive.cli.print.header=true; // 列印列名
set hive.cli.print.row.to.vertical=true; // 開啟行轉列功能, 前提必須開啟列印列名功能
set hive.cli.print.row.to.vertical.num=1; // 設定每行顯示的列數

2.使用過程中出錯採用：
hive -hiveconf hive.root.logger=DEBUG,console //重啟除錯。

3. hive的三種啟動方式區別：
1，hive 命令列模式，直接輸入/hive/bin/hive的執行程式，或者輸入 hive –service cli
用於linux平臺命令列查詢，查詢語句基本跟mysql查詢語句類似
2，hive web介面的啟動方式，hive –service hwi
用於通過瀏覽器來訪問hive，感覺沒多大用途
3，hive 遠端服務 (埠號10000) 啟動方式，nohup hive –service hiveserver &
用java等程式實現通過jdbc等驅動的訪問hive就用這種起動方式了，這個是程式設計師最需要的方式了
啟動hive service ：$HIVE_HOME/bin/hive --service hiveserver 10001 >/dev/null 2>/dev/null &

4. hive插入的2中方式：
基本的插入語法：
INSERT OVERWRITE TABLE tablename [PARTITON(partcol1=val1,partclo2=val2)]select_statement FROM from_statement
insert overwrite table test_insert select * from test_table;
對多個表進行插入操作：
FROM fromstatte
INSERT OVERWRITE TABLE tablename1 [PARTITON(partcol1=val1,partclo2=val2)]select_statement1
INSERT OVERWRITE TABLE tablename2 [PARTITON(partcol1=val1,partclo2=val2)]select_statement2

5.新增metastore啟動指令碼bin/hive-metastore.sh
#!/bin/sh
nohup ./hive --service metastore >> metastore.log 2>&1 &
echo $! > hive-metastore.pid

新增hive server啟動指令碼bin/hive-server.sh
nohup ./hive --service hiveserver >> hiveserver.log 2>&1 &
echo $! > hive-server.pid

啟動metastore和hive server
./hive-metastore.sh
./hive-server.sh

轉：http://blog.csdn.net/wulantian/article/details/38112359

三、hive的具體練習：（以下4個目標）
1. 第一普通的hdfs檔案能匯入到hive中，以供我們查詢。
2. 第二hbase中的表，能匯入hive中，以供我們查詢。
3. 第三mysql中的表，能匯入hive中，以供我們查詢。
4. hive中的各種查詢分析結果，能匯入到mysql當中，以後頁面展示。

本文是第一個目標：
第一普通的hdfs檔案能匯入到hive中，以供我們查詢。同時，查詢的結果能儲存到一下3個地方：
1.將select的結果放到一個的的表格中（首先要用create table建立新的表格）
2.將select的結果放到本地檔案系統中
3.將select的結果放到hdfs檔案系統中

下面具體目標分別測試：
1. 普通的hdfs檔案匯入到hive中。
建立一個root目錄下一個普通檔案：
[[email protected] ~]# cat hello.txt
# This is a text txt
# by coco
# 2014-07-18
在hive中匯入該檔案：
hive> create table pokes(foo int,bar string);
OK
Time taken: 0.068 seconds
hive> load data local inpath '/root/hello.txt' overwrite into table pokes;
Copying data from file:/root/hello.txt
Copying file: file:/root/hello.txt
Loading data to table default.pokes
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted hdfs://db96:9000/user/hive/warehouse/pokes
Table default.pokes stats: [numFiles=1, numRows=0, totalSize=59, rawDataSize=0]
OK
Time taken: 0.683 seconds
hive> select * from pokes;
OK
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
Time taken: 0.237 seconds, Fetched: 5 row(s)
hive>
hive> load data local inpath '/root/hello.txt' overwrite into table test;
Copying data from file:/root/hello.txt
Copying file: file:/root/hello.txt
Loading data to table default.test
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted hdfs://db96:9000/hive/warehousedir/test
Failed with exception Unable to alter table.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
hive> show tables;
OK
hivetest
pokes
test
test3
Time taken: 1.045 seconds, Fetched: 4 row(s)
hive> select * from test;
OK
# This is a text txt NULL
# by coco NULL
# 2014-07-18 NULL
NULL
hello world!! NULL
Time taken: 1.089 seconds, Fetched: 5 row(s)
從上面看匯入成功，但是查詢的都是null，那是因為沒有加分隔.test表預設的有terminated by '\t'
lines terminated by '\n' 分隔符，所以儘管有報錯，資料也是插入的。
正確的匯入語法為：
create table aaa(time string,myname string,yourname string) row format delimited
fields terminated by '\t' lines terminated by '\n' stored as textfile
hive> create table aaa(time string,myname string,yourname string) row format delimited
> fields terminated by '\t' lines terminated by '\n' stored as textfile;
OK
Time taken: 1.011 seconds
hive> load data local inpath '/root/aaaa.txt' overwrite
> into table aaa;
Copying data from file:/root/aaaa.txt
Copying file: file:/root/aaaa.txt
Loading data to table default.aaa
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted hdfs://db96:9000/hive/warehousedir/aaa
[Warning] could not update stats.
OK
Time taken: 2.686 seconds
hive> select * from aaa;
OK
20140723,yting,xmei NULL NULL
Time taken: 0.054 seconds, Fetched: 1 row(s)

2. 查詢結果匯出來。
從hive中把表中的資料匯出來，儲存成文字型別。
先檢索索要的結果：
hive> select time from aaa;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1405999790746_0002, Tracking URL = http://db96:8088/proxy/application_1405999790746_0002/
Kill Command = /usr/local/hadoop//bin/hadoop job -kill job_1405999790746_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-07-23 16:28:51,690 Stage-1 map = 0%, reduce = 0%
2014-07-23 16:29:02,457 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.33 sec
MapReduce Total cumulative CPU time: 1 seconds 330 msec
Ended Job = job_1405999790746_0002
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 1.33 sec HDFS Read: 221 HDFS Write: 20 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 330 msec
OK
20140723,yting,xmei
Time taken: 26.281 seconds, Fetched: 1 row(s)
將查詢結果輸出至本地目錄
hive> insert overwrite local directory '/tmp' select a.time from aaa a;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1405999790746_0004, Tracking URL = http://db96:8088/proxy/application_1405999790746_0004/
Kill Command = /usr/local/hadoop//bin/hadoop job -kill job_1405999790746_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-07-23 16:34:28,474 Stage-1 map = 0%, reduce = 0%
2014-07-23 16:34:35,128 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.27 sec
MapReduce Total cumulative CPU time: 1 seconds 270 msec
Ended Job = job_1405999790746_0004
Copying data to local directory /tmp
Copying data to local directory /tmp
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 1.27 sec HDFS Read: 221 HDFS Write: 20 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 270 msec
OK
Time taken: 21.943 seconds
可以看到/tmp下確實有一個檔案，000000_0，該檔案的內容為，我們查詢看到的內容。

[email protected] tmp]# ll
總用量 4
-rw-r--r-- 1 root root 20 7月 23 16:34 000000_0
[[email protected] tmp]# vim 000000_0

20140723,yting,xmei
~
很多時候，我們在hive中執行select語句，希望將最終的結果儲存到本地檔案或者儲存到hdfs系統中
或者儲存到一個新的表中，hive提供了方便的關鍵詞，來實現上面所述的功能。
1.將select的結果放到一個的的表格中（首先要用create table建立新的表格）
insert overwrite table test select uid,name from test2;
2.將select的結果放到本地檔案系統中
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/reg_3' SELECT a.* FROM events a;
3.將select的結果放到hdfs檔案系統中
INSERT OVERWRITE DIRECTORY '/tmp/hdfs_out' SELECT a.* FROM invites a WHERE a.ds='<DATE>';

以上，我們實現了把普通本地的文字檔案匯入到hive中，並能實現相關的查詢，並把查詢結果匯出到3個不同的地方。

具體示例：
hive> select a.time from aaa a;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1405999790746_0005, Tracking URL = http://db96:8088/proxy/application_1405999790746_0005/
Kill Command = /usr/local/hadoop//bin/hadoop job -kill job_1405999790746_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-07-23 16:47:42,295 Stage-1 map = 0%, reduce = 0%
2014-07-23 16:47:49,567 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.19 sec
MapReduce Total cumulative CPU time: 1 seconds 190 msec
Ended Job = job_1405999790746_0005
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 1.19 sec HDFS Read: 221 HDFS Write: 20 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 190 msec
OK
a.time
20140723,yting,xmei
Time taken: 21.155 seconds, Fetched: 1 row(s)
hive> create table jieguo(content string);
OK
Time taken: 2.424 seconds
hive> insert overwrite table jieguo
> select a.time from aaa a;
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1405999790746_0006, Tracking URL = http://db96:8088/proxy/application_1405999790746_0006/
Kill Command = /usr/local/hadoop//bin/hadoop job -kill job_1405999790746_0006
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-07-23 16:49:50,689 Stage-1 map = 0%, reduce = 0%
2014-07-23 16:49:57,329 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.3 sec
MapReduce Total cumulative CPU time: 1 seconds 300 msec
Ended Job = job_1405999790746_0006
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://db96:9000/hive/scratchdir/hive_2014-07-23_16-49-36_884_4745480606977792448-1/-ext-10000
Loading data to table default.jieguo
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted hdfs://db96:9000/hive/warehousedir/jieguo
Failed with exception Unable to alter table.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 1.3 sec HDFS Read: 221 HDFS Write: 90 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 300 msec
hive> show tables;
OK
tab_name
aaa
hello
hivetest
jieguo
pokes
test
test3
Time taken: 1.03 seconds, Fetched: 7 row(s)
hive> select * from jieguo;
OK
jieguo.content
20140723,yting,xmei
Time taken: 1.043 seconds, Fetched: 1 row(s)

轉：http://blog.csdn.net/wulantian/article/details/38111701

HIVE與mysql的關係 hive常用命令整理 hive與hdfs整合過程

HIVE與mysql的關係 hive常用命令整理 hive與hdfs整合過程

mysql常用命令整理

mysql備份與還原資料庫的常用命令。

[轉]備份與還原mysql 資料庫的常用命令。

HBASE與hive對比使用以及HBASE常用shell操作。與sqoop的集成

Git使用：安裝，使用及常用命令整理

git常用命令整理

salt 常用命令整理

Redis學習筆記（三）常用命令整理

linux常用命令整理（五）：shell基礎

Linux常用命令整理

mysql線上操作常用命令

hive的mysql作元數據的hive-site.xml配置

四、mysql數據常用命令

redis常用命令整理

linux系統配置常用命令整理

windows常用命令整理

Linux常用命令簡述--cat與tac

mysql生產環境常用命令

HDFS常用命令整理

HIVE與mysql的關係 hive常用命令整理 hive與hdfs整合過程

相關推薦