我們經常利用Hive完成一些資料報表統計的需求，此時我們在特定場景中需要對Hive查詢的結果做匯出，

這樣就延伸出了幾種查詢結果匯出的方法，我們這裡做一下總結。

1.利用beeline 執行查詢，將輸出重定向的方式

2.利用insert overwrite 的方式，將查詢結果匯出hdfs或者本地

第一步我們看下主體的查詢語句：

use default;


SELECT media_info.media_name, media_tmp.* , activity_info.*, camp_media_tmp.*
FROM 
(
	SELECT
        	activity.id,
        	activity.activity_name,
       		advertiser.advertiser_name,
        	agency.agency_name
	FROM promotion_activity_mysql AS activity
	JOIN admin_advertiser_mysql AS advertiser ON activity.advertiser_id = advertiser.id
	JOIN admin_agency_mysql AS agency ON advertiser.agency_id = agency.id
) AS activity_info
JOIN 
(
	SELECT
        	mid.mediaid     ,
       		mid.campaignid  ,
       		SUM(mid.campimp) as camp_imp    ,
        	SUM(mid.campclick) as camp_click        ,
        	SUM(mid.campclickcost) as click_cost
	FROM
        	clickcube_mid AS  mid
	WHERE   mid.day >= '2018-06-01' AND mid.day <= '2018-06-30'
	GROUP BY  mid.mediaid, mid.campaignid
) AS camp_media_tmp ON activity_info.id = camp_media_tmp.campaignid 
JOIN
(
	SELECT
        	'2018-06'       ,
        	mid.mediaid     ,
        	SUM(mid.mediaimp) AS  media_imp_total   ,
        	SUM(mid.mediaclick) AS media_clk_total      ,
        	SUM(mid.mediaclickcost) AS media_cost
	FROM
        	clickcube_mid AS  mid
	WHERE   mid.day >= '2018-06-01' AND mid.day <= '2018-06-30'
	GROUP BY  mid.mediaid
) AS media_tmp ON  camp_media_tmp.mediaid = media_tmp.mediaid
JOIN 
admin_media_mysql AS media_info ON media_info.id = media_tmp.mediaid 
ORDER BY media_tmp.mediaid, camp_media_tmp.campaignid 
;
 

1.首先我們看下beeline執行查詢，檔案重定向的方式

缺點：該種將資料匯出的方式，其實跟在控制檯將查詢結果輸出並沒有什麼區別。

輸出會包含很多冗餘的資訊

首先我們需要對beeline 的引數較為了解：

傳送門：

Hive _ Hive2 新版連線工具 beeline 詳解

這裡我們需要注意一個問題，hive查詢預設不顯示錶頭的，我們需要開啟顯示錶頭的開關

我們將這個操作寫到 **.init 檔案中

final_to_hdfs_201806.init

SET hive.cli.print.header=true;

這裡我們就可以利用檔案中重定向了,我們將beeline 執行的過程寫到shell 指令碼中 .

final_to_file.sh

#!/bin/bash


beeline -u jdbc:hive2://10.180.0.26:10000 -n cloudera-scm -f final_to_file_201806_report.hql -i final_to_hdfs_201806.init > final_report_file.txt

看下最終執行的結果

[[email protected] 20180703_media_agency_job]$ head -n 10 final_report_file.txt 
0: jdbc:hive2://10.180.0.26:10000> 
0: jdbc:hive2://10.180.0.26:10000> SET hive.cli.print.header=true;
0: jdbc:hive2://10.180.0.26:10000> 
0: jdbc:hive2://10.180.0.26:10000> 
0: jdbc:hive2://10.180.0.26:10000> 
0: jdbc:hive2://10.180.0.26:10000> use default;
0: jdbc:hive2://10.180.0.26:10000> 
0: jdbc:hive2://10.180.0.26:10000> SELECT media_info.media_name, media_tmp.* , activity_info.*, camp_media_tmp.*
. . . . . . . . . . . . . . . . .> FROM 
. . . . . . . . . . . . . . . . .> (
 

2.利用insert overwrite 的方式，將查詢結果匯出hdfs或者本地

缺點： 該種方式並沒有資料表頭的資訊，需要手動增加表頭

首先我們看下對Hive 資料匯出語句 INSERT OVERWRITE 的介紹：

Standard syntax:
INSERT OVERWRITE [LOCAL] DIRECTORY directory1
  [ROW FORMAT row_format] [STORED AS file_format] (Note: Only available starting with Hive 0.11.0)
  SELECT ... FROM ...
 
Hive extension (multiple inserts):
FROM from_statement
INSERT OVERWRITE [LOCAL] DIRECTORY directory1 select_statement1
[INSERT OVERWRITE [LOCAL] DIRECTORY directory2 select_statement2] ...
 
  
row_format
  : DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char]
        [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char]
        [NULL DEFINED AS char] (Note: Only available starting with Hive 0.13)

可以該種方式的支援比較完善

首先匯出的資料格式是可選的：

[STORED AS file_format] (Note: Only available starting with Hive 0.11.0)

也就是說匯出的資料格式是可選的，到目前為止可選的資料檔案型別主要為以下幾種檔案：

file_format:  

  : SEQUENCEFILE  

    含有資料的元資訊，可分塊，資料按行組織起來

  | TEXTFILE  

    TEXTFILE 最原始的文字檔案

  | RCFILE     (Note: Only available starting with Hive 0.6.0)  

    資料按列組織起來

  | ORC        (Note: Only available starting with Hive 0.11.0)  

  | AVRO       (Note: Only available starting with Hive 0.14.0)  

  | PARQUET

     資料按行組織起來

這裡我們存為textfile

示例：

STORED AS TEXTFILE

其次匯出的欄位分隔符是可以自己設定的，   

Hive 中欄位之間的分隔符預設是 ^A (ascii碼是\00001),

不便於修改檢視，我們將分隔符改為 , (逗號)

示例：

ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','

最後查詢的資料檔案的位置是可以指定的，可以存放在執行查詢的機器的本地磁碟上，也可以存放在hdfs上，這裡我們指定hdfs.

INSERT OVERWRITE [LOCAL] DIRECTORY directory1

示例：

INSERT OVERWRITE directory '/tmp/201806_report'

最終的查詢語句

use default;

INSERT OVERWRITE directory '/tmp/201806_report'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
-- STORED AS SEQUENCEFILE
SELECT media_info.media_name, media_tmp.* , activity_info.*, camp_media_tmp.*
FROM 
(
	SELECT
        	activity.id,
        	activity.activity_name,
       		advertiser.advertiser_name,
        	agency.agency_name
	FROM promotion_activity_mysql AS activity
	JOIN admin_advertiser_mysql AS advertiser ON activity.advertiser_id = advertiser.id
	JOIN admin_agency_mysql AS agency ON advertiser.agency_id = agency.id
) AS activity_info
JOIN 
(
	SELECT
        	mid.mediaid     ,
       		mid.campaignid  ,
       		SUM(mid.campimp) as camp_imp    ,
        	SUM(mid.campclick) as camp_click        ,
        	SUM(mid.campclickcost) as click_cost
	FROM
        	clickcube_mid AS  mid
	WHERE   mid.day >= '2018-06-01' AND mid.day <= '2018-06-30'
	GROUP BY  mid.mediaid, mid.campaignid
) AS camp_media_tmp ON activity_info.id = camp_media_tmp.campaignid 
JOIN
(
	SELECT
        	'2018-06'       ,
        	mid.mediaid     ,
        	SUM(mid.mediaimp) AS  media_imp_total   ,
        	SUM(mid.mediaclick) AS media_clk_total      ,
        	SUM(mid.mediaclickcost) AS media_cost
	FROM
        	clickcube_mid AS  mid
	WHERE   mid.day >= '2018-06-01' AND mid.day <= '2018-06-30'
	GROUP BY  mid.mediaid
) AS media_tmp ON  camp_media_tmp.mediaid = media_tmp.mediaid
JOIN 
admin_media_mysql AS media_info ON media_info.id = media_tmp.mediaid 
ORDER BY media_tmp.mediaid, camp_media_tmp.campaignid 
;

我們將執行過程寫入指令碼中

#!/bin/bash


beeline -u jdbc:hive2://10.180.0.26:10000 -n cloudera-scm -f final_to_hdfs_201806_report.hql -i final_to_hdfs_201806.init 

最終的執行結果：

我們檢視下執行結果：