Hive靜態分割槽表

阿新 • • 發佈：2019-01-28

Hive的分割槽表分為動態分割槽和靜態分割槽，分割槽表的使用能夠為巨量表查詢效能的提高提供幫助。
靜態分割槽在資料載入前需要事先將分割槽建好，使用起來稍顯複雜，而動態表可以根據資料自動建立分割槽，但同時花費了巨大的效能代價。如果分割槽是可以確定的話，一定不要用動態分割槽，動態分割槽的值是在reduce執行階段確定的；也就是會把所有的記錄distribute by。可想而知表記錄非常大的話，只有一個reduce去處理，那簡直是瘋狂的。如果這個值唯一或者事先已經知道，比如按天分割槽(i_date=20151105)那就用靜態分割槽吧。靜態分割槽在編譯階段已經確定，不需要reduce處理。所以，在分割槽表的使用上，一般建議使用靜態分割槽。

partition必須在表定義時建立。
1、單分割槽建表語句

CREATE TABLE `base_order_partition`(
  `order_id` bigint, 
  `order_date` int, 
  `order_mo` int, 
  `status` tinyint, 
  `status_desc` varchar(12), 
  `food_num` int, 
  `food_amount` float, 
  `order_amount` float)
PARTITIONED BY ( 
  `order_month` int)
ROW FORMAT DELIMITED 
  FIELDS TERMINATED BY '$' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://Master:9000/home/spark/opt/data_dir/hive/warehouse/59store.db/base_order_partition'
TBLPROPERTIES (
  'transient_lastDdlTime'='1446600982');

2、雙分割槽建表語句

CREATE TABLE `base_order_partition2`(
  `order_id` bigint, 
  `order_date` int, 
  `order_mo` int, 
  `status` tinyint, 
  `status_desc` varchar(12), 
  `food_num` int, 
  `food_amount` float, 
  `order_amount` float)
PARTITIONED BY ( 
  `order_month` int, 
  `order_date1` int)
ROW FORMAT DELIMITED 
  FIELDS TERMINATED BY '$' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://Master:9000/home/spark/opt/data_dir/hive/warehouse/59store.db/base_order_partition2'
TBLPROPERTIES (
  'transient_lastDdlTime'='1446620191');

先以order_month為資料夾，再以order_date1子資料夾區分。

3、新增分割槽表語法（表已建立，在此基礎上新增分割槽）

單分割槽表：
alter table base_order_partition add partition (order_month=201501);
alter table base_order_partition add partition (order_month=201502);
alter table base_order_partition add partition (order_month=201503);
alter table base_order_partition add partition (order_month=201504);
alter table base_order_partition add partition (order_month=201505);
alter table base_order_partition add partition (order_month=201506);
alter table base_order_partition add partition (order_month=201507);
alter table base_order_partition add partition (order_month=201508);
alter table base_order_partition add partition (order_month=201509);
alter table base_order_partition add partition (order_month=201510);
alter table base_order_partition add partition (order_month=201511);
alter table base_order_partition add partition (order_month=201512);
雙分割槽表：
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151001);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151002);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151003);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151004);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151005);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151006);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151007);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151008);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151009);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151010);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151011);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151012);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151013);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151014);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151015);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151016);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151017);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151018);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151019);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151020);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151021);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151022);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151023);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151024);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151025);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151026);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151027);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151028);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151029);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151030);
alter table base_order_partition2 add partition (order_month=201510,order_date1=20151031);
hive> alter table base_order_partition2 add partition (order_month=201510,order_date1=20151032);

4、刪除分割槽語法
ALTER TABLE table_name DROP
partition_spec, partition_spec,...

使用者可以用 ALTER TABLE DROP PARTITION 來刪除分割槽。分割槽的元資料和資料將被一併刪除。

hive> alter table base_order_partition2 drop partition (order_month=201510,order_date1=20151032);

5、資料載入進分割槽表中語法
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
例：
LOAD DATA INPATH '/home/spark/opt/data_dir/pv.txt' INTO TABLE day_hour_table PARTITION(dt='2008-08- 08', hour='08'); LOAD DATA local INPATH '/home/spark/opt/data_dir/hua/*' INTO TABLE day_hour partition(dt='2010-07- 07');
當資料被載入至表中時，不會對資料進行任何轉換。Load操作只是將資料複製至Hive表對應的位置。資料載入時在表下自動建立一個目錄。

表對錶方式：

from base_order_partition
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151001)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151001
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151002)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151002
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151003)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151003
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151004)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151004
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151005)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151005
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151006)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151006
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151007)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151007
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151008)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151008
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151009)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151009
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151010)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151010
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151011)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151011
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151012)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151012
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151013)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151013
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151014)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151014
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151015)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151015
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151016)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151016
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151017)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151017
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151018)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151018
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151019)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151019
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151020)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151020
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151021)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151021
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151022)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151022
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151023)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151023
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151024)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151024
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151025)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151025
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151026)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151026
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151027)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151027
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151028)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151028
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151029)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151029
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151030)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151030
insert overwrite table base_order_partition2 partition (order_month=201510,order_date1=20151031)
select `order_id`,`order_date`,`order_mo`,`status`,`status_desc`,`food_num`,`food_amount`,`order_amount`, where order_month=201510 and order_date=20151031
;

6、基於分割槽的查詢的語句

SELECT * FROM base_order_partition WHERE order_month=201511;

7、檢視分割槽語句

hive> show partitions base_order_partition;
OK
order_month=201501
order_month=201502
order_month=201503
order_month=201504
order_month=201505
order_month=201506
order_month=201507
order_month=201508
order_month=201509
order_month=201510
order_month=201511
order_month=201512
Time taken: 0.076 seconds, Fetched: 12 row(s)

Hive靜態分割槽表

Hive靜態分割槽表

Hive中靜態分割槽表 & 動態分割槽表

Hive: 建立分割槽表（partition表）及分割槽表匯入csv文字檔案資料

hive新建分割槽表

大資料開發之Hive篇----分割槽表的構建

Hive學習筆記 --- Hive建立分割槽表

通過spark sql建立HIVE的分割槽表

Hive優化--分割槽表與分桶表

hive修改分割槽表

Hive中分割槽表及陷阱

Hive外部分割槽表載入flume打到hdfs上檔案，讀不到.tmp檔案

hive內部分割槽表(ORC格式)新增欄位後出現的問題

Hive靜態分割槽和動態分割槽

hive中簡單介紹分割槽表(partition table)，含動態分割槽(dynamic partition)與靜態分割槽(static partition)

hive根據現有資料表建立分割槽表,進行動態或靜態分割槽插入資料

hive 分割槽partition表建立資料匯入（動態分割槽插入、靜態分割槽插入、動靜態混合插入）

hive：普通表外部表分割槽表

Hive內部表、外部表、分割槽表以及外部分割槽表建立以及匯入資料例項講解

Hive的靜態分割槽與動態分割槽應用場景

hive中的靜態分割槽與動態分割槽

Hive靜態分割槽表

相關推薦