Hive常用函式大全（二）（視窗函式、分析函式、增強group）
阿新 • • 發佈：2018-12-14
視窗函式與分析函式
應用場景： （1）用於分割槽排序 （2）動態Group By （3）Top N （4）累計計算 （5）層次查詢
視窗函式
FIRST_VALUE：取分組內排序後，截止到當前行，第一個值 LAST_VALUE： 取分組內排序後，截止到當前行，最後一個值 LEAD(col,n,DEFAULT) ：用於統計視窗內往下第n行值。第一個引數為列名，第二個引數為往下第n行（可選，預設為1），第三個引數為預設值（當往下第n行為NULL時候，取預設值，如不指定，則為NULL） LAG(col,n,DEFAULT) ：與lead相反，用於統計視窗內往上第n行值。第一個引數為列名，第二個引數為往上第n行（可選，預設為1），第三個引數為預設值（當往上第n行為NULL時候，取預設值，如不指定，則為NULL）
OVER從句
1、使用標準的聚合函式COUNT、SUM、MIN、MAX、AVG
2、使用PARTITION BY語句，使用一個或者多個原始資料型別的列
3、使用PARTITION BY與ORDER BY語句，使用一個或者多個數據型別的分割槽或者排序列
4、使用視窗規範，視窗規範支援以下格式：
(ROWS | RANGE) BETWEEN (UNBOUNDED | [num]) PRECEDING AND ([num] PRECEDING | CURRENT ROW | (UNBOUNDED | [num]) FOLLOWING) (ROWS | RANGE) BETWEEN CURRENT ROW AND (CURRENT ROW | (UNBOUNDED | [num]) FOLLOWING) (ROWS | RANGE) BETWEEN [num] FOLLOWING AND (UNBOUNDED | [num]) FOLLOWING
- 1
- 2
- 3
當ORDER BY後面缺少視窗從句條件，視窗規範預設是 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.
當ORDER BY和視窗從句都缺失, 視窗規範預設是 ROW BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.
OVER從句支援以下函式， 但是並不支援和視窗一起使用它們。
Ranking函式: Rank, NTile, DenseRank, CumeDist, PercentRank.Lead 和 Lag 函式.
分析函式
ROW_NUMBER() 從1開始，按照順序，生成分組內記錄的序列,比如，按照pv降序排列，生成分組內每天的pv名次,ROW_NUMBER()的應用場景非常多，再比如，獲取分組內排序第一的記錄;獲取一個session中的第一條refer等。 RANK() 生成資料項在分組中的排名，排名相等會在名次中留下空位 DENSE_RANK() 生成資料項在分組中的排名，排名相等會在名次中不會留下空位 CUME_DIST 小於等於當前值的行數/分組內總行數。比如，統計小於等於當前薪水的人數，所佔總人數的比例 PERCENT_RANK 分組內當前行的RANK值-1/分組內總行數-1 NTILE(n) 用於將分組資料按照順序切分成n片，返回當前切片值，如果切片不均勻，預設增加第一個切片的分佈。NTILE不支援ROWS BETWEEN，比如 NTILE(2) OVER(PARTITION BY cookieid ORDER BY createtime ROWS BETWEEN 3 PRECEDING AND CURRENT ROW)。
Hive2.1.0及以後支援Distinct
在聚合函式（SUM, COUNT and AVG）中，支援distinct，但是在ORDER BY 或者 視窗限制不支援。
COUNT(DISTINCT a) OVER (PARTITION BY c)- 1
Hive 2.2.0中在使用ORDER BY和視窗限制時支援distinct
COUNT(DISTINCT a) OVER (PARTITION BY c ORDER BY d ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)- 1
Hive2.1.0及以後支援在OVER從句中支援聚合函式
SELECT rank() OVER (ORDER BY sum(b))
FROM T
GROUP BY a;- 1
- 2
- 3
測試資料集：
## COUNT、SUM、MIN、MAX、AVG
select
user_id,
user_type,
sales,
--預設為從起點到當前行
sum(sales) OVER(PARTITION BY user_type ORDER BY sales asc) AS sales_1,
--從起點到當前行，結果與sales_1不同。
sum(sales) OVER(PARTITION BY user_type ORDER BY sales asc ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS sales_2,
--當前行+往前3行
sum(sales) OVER(PARTITION BY user_type ORDER BY sales asc ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS sales_3,
--當前行+往前3行+往後1行
sum(sales) OVER(PARTITION BY user_type ORDER BY sales asc ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) AS sales_4,
--當前行+往後所有行
sum(sales) OVER(PARTITION BY user_type ORDER BY sales asc ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS sales_5,
--分組內所有行
SUM(sales) OVER(PARTITION BY user_type) AS sales_6
from
order_detail
order by
user_type,
sales,
user_id
+----------+------------+--------+----------+----------+----------+----------+----------+----------+--+
| user_id | user_type | sales | sales_1 | sales_2 | sales_3 | sales_4 | sales_5 | sales_6 |
+----------+------------+--------+----------+----------+----------+----------+----------+----------+--+
| liiu | new | 1 | 2 | 2 | 2 | 4 | 22 | 23 |
| qibaqiu | new | 1 | 2 | 1 | 1 | 2 | 23 | 23 |
| zhangsa | new | 2 | 4 | 4 | 4 | 7 | 21 | 23 |
| wanger | new | 3 | 7 | 7 | 7 | 12 | 19 | 23 |
| lilisi | new | 5 | 17 | 17 | 15 | 21 | 11 | 23 |
| qishili | new | 5 | 17 | 12 | 11 | 16 | 16 | 23 |
| wutong | new | 6 | 23 | 23 | 19 | 19 | 6 | 23 |
| lisi | old | 1 | 1 | 1 | 1 | 3 | 6 | 6 |
| wangshi | old | 2 | 3 | 3 | 3 | 6 | 5 | 6 |
| liwei | old | 3 | 6 | 6 | 6 | 6 | 3 | 6 |
+----------+------------+--------+----------+----------+----------+----------+----------+----------+--+
注意:
結果和ORDER BY相關,預設為升序
如果不指定ROWS BETWEEN,預設為從起點到當前行;
如果不指定ORDER BY，則將分組內所有值累加;
關鍵是理解ROWS BETWEEN含義,也叫做WINDOW子句：
PRECEDING：往前
FOLLOWING：往後
CURRENT ROW：當前行
UNBOUNDED：無界限（起點或終點）
UNBOUNDED PRECEDING：表示從前面的起點
UNBOUNDED FOLLOWING：表示到後面的終點
其他COUNT、AVG，MIN，MAX，和SUM用法一樣。- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
## first_value與last_value
select
user_id,
user_type,
ROW_NUMBER() OVER(PARTITION BY user_type ORDER BY sales) AS row_num,
first_value(user_id) over (partition by user_type order by sales desc) as max_sales_user,
first_value(user_id) over (partition by user_type order by sales asc) as min_sales_user,
last_value(user_id) over (partition by user_type order by sales desc) as curr_last_min_user,
last_value(user_id) over (partition by user_type order by sales asc) as curr_last_max_user
from
order_detail;
+----------+------------+----------+-----------------+-----------------+---------------------+---------------------+--+
| user_id | user_type | row_num | max_sales_user | min_sales_user | curr_last_min_user | curr_last_max_user |
+----------+------------+----------+-----------------+-----------------+---------------------+---------------------+--+
| wutong | new | 7 | wutong | qibaqiu | wutong | wutong |
| lilisi | new | 6 | wutong | qibaqiu | qishili | lilisi |
| qishili | new | 5 | wutong | qibaqiu | qishili | lilisi |
| wanger | new | 4 | wutong | qibaqiu | wanger | wanger |
| zhangsa | new | 3 | wutong | qibaqiu | zhangsa | zhangsa |
| liiu | new | 2 | wutong | qibaqiu | qibaqiu | liiu |
| qibaqiu | new | 1 | wutong | qibaqiu | qibaqiu | liiu |
| liwei | old | 3 | liwei | lisi | liwei | liwei |
| wangshi | old | 2 | liwei | lisi | wangshi | wangshi |
| lisi | old | 1 | liwei | lisi | lisi | lisi |
+----------+------------+----------+-----------------+-----------------+---------------------+---------------------+--+
## lead與lag
select
user_id,device_id,
lead(device_id) over (order by sales) as default_after_one_line,
lag(device_id) over (order by sales) as default_before_one_line,
lead(device_id,2) over (order by sales) as after_two_line,
lag(device_id,2,'abc') over (order by sales) as before_two_line
from
order_detail;
+----------+-------------+-------------------------+--------------------------+-----------------+------------------+--+
| user_id | device_id | default_after_one_line | default_before_one_line | after_two_line | before_two_line |
+----------+-------------+-------------------------+--------------------------+-----------------+------------------+--+
| qibaqiu | fds | fdsfagwe | NULL | 543gfd | abc |
| liiu | fdsfagwe | 543gfd | fds | f332 | abc |
| lisi | 543gfd | f332 | fdsfagwe | dfsadsa323 | fds |
| wangshi | f332 | dfsadsa323 | 543gfd | hfd | fdsfagwe |
| zhangsa | dfsadsa323 | hfd | f332 | 65ghf | 543gfd |
| liwei | hfd | 65ghf | dfsadsa323 | fds | f332 |
| wanger | 65ghf | fds | hfd | dsfgg | dfsadsa323 |
| qishili | fds | dsfgg | 65ghf | 543gdfsd | hfd |
| lilisi | dsfgg | 543gdfsd | fds | NULL | 65ghf |
| wutong | 543gdfsd | NULL | dsfgg | NULL | fds |
+----------+-------------+-------------------------+--------------------------+-----------------+------------------+--+
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
## RANK、ROW_NUMBER、DENSE_RANK
select
user_id,user_type,sales,
RANK() over (partition by user_type order by sales desc) as r,
ROW_NUMBER() over (partition by user_type order by sales desc) as rn,
DENSE_RANK() over (partition by user_type order by sales desc) as dr
from
order_detail;
+----------+------------+--------+----+-----+-----+--+
| user_id | user_type | sales | r | rn | dr |
+----------+------------+--------+----+-----+-----+--+
| wutong | new | 6 | 1 | 1 | 1 |
| qishili | new | 5 | 2 | 2 | 2 |
| lilisi | new | 5 | 2 | 3 | 2 |
| wanger | new | 3 | 4 | 4 | 3 |
| zhangsa | new | 2 | 5 | 5 | 4 |
| qibaqiu | new | 1 | 6 | 6 | 5 |
| liiu | new | 1 | 6 | 7 | 5 |
| liwei | old | 3 | 1 | 1 | 1 |
| wangshi | old | 2 | 2 | 2 | 2 |
| lisi | old | 1 | 3 | 3 | 3 |
+----------+------------+--------+----+-----+-----+--+ - 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
## NTILE
select
user_type,sales,
--分組內將資料分成2片
NTILE(2) OVER(PARTITION BY user_type ORDER BY sales) AS nt2,
--分組內將資料分成3片
NTILE(3) OVER(PARTITION BY user_type ORDER BY sales) AS nt3,
--分組內將資料分成4片
NTILE(4) OVER(PARTITION BY user_type ORDER BY sales) AS nt4,
--將所有資料分成4片
NTILE(4) OVER(ORDER BY sales) AS all_nt4
from
order_detail
order by
user_type,
sales
+------------+--------+------+------+------+----------+--+
| user_type | sales | nt2 | nt3 | nt4 | all_nt4 |
+------------+--------+------+------+------+----------+--+
| new | 1 | 1 | 1 | 1 | 1 |
| new | 1 | 1 | 1 | 1 | 1 |
| new | 2 | 1 | 1 | 2 | 2 |
| new | 3 | 1 | 2 | 2 | 3 |
| new | 5 | 2 | 2 | 3 | 4 |
| new | 5 | 2 | 3 | 3 | 3 |
| new | 6 | 2 | 3 | 4 | 4 |
| old | 1 | 1 | 1 | 1 | 1 |
| old | 2 | 1 | 2 | 2 | 2 |
| old | 3 | 2 | 3 | 3 | 2 |
+------------+--------+------+------+------+----------+--+
求取sale前20%的使用者ID
select
user_id
from
(
select
user_id,
NTILE(5) OVER(ORDER BY sales desc) AS nt
from
order_detail
)A
where nt=1;- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
## CUME_DIST、PERCENT_RANK
select
user_id,user_type,sales,
--沒有partition,所有資料均為1組
CUME_DIST() OVER(ORDER BY sales) AS cd1,
--按照user_type進行分組
CUME_DIST() OVER(PARTITION BY user_type ORDER BY sales) AS cd2
from
order_detail;
+----------+------------+--------+------+----------------------+--+
| user_id | user_type | sales | cd1 | cd2 |
+----------+------------+--------+------+----------------------+--+
| liiu | new | 1 | 0.3 | 0.2857142857142857 |
| qibaqiu | new | 1 | 0.3 | 0.2857142857142857 |
| zhangsa | new | 2 | 0.5 | 0.42857142857142855 |
| wanger | new | 3 | 0.7 | 0.5714285714285714 |
| lilisi | new | 5 | 0.9 | 0.8571428571428571 |
| qishili | new | 5 | 0.9 | 0.8571428571428571 |
| wutong | new | 6 | 1.0 | 1.0 |
| lisi | old | 1 | 0.3 | 0.3333333333333333 |
| wangshi | old | 2 | 0.5 | 0.6666666666666666 |
| liwei | old | 3 | 0.7 | 1.0 |
+----------+------------+--------+------+----------------------+--+
select
user_type,sales
--分組內總行數
SUM(1) OVER(PARTITION BY user_type) AS s,
--RANK值
RANK() OVER(ORDER BY sales) AS r,
PERCENT_RANK() OVER(ORDER BY sales) AS pr,
--分組內
PERCENT_RANK() OVER(PARTITION BY user_type ORDER BY sales) AS prg
from
order_detail;
+----+-----+---------------------+---------------------+--+
| s | r | pr | prg |
+----+-----+---------------------+---------------------+--+
| 7 | 1 | 0.0 | 0.0 |
| 7 | 1 | 0.0 | 0.0 |
| 7 | 4 | 0.3333333333333333 | 0.3333333333333333 |
| 7 | 6 | 0.5555555555555556 | 0.5 |
| 7 | 8 | 0.7777777777777778 | 0.6666666666666666 |
| 7 | 8 | 0.7777777777777778 | 0.6666666666666666 |
| 7 | 10 | 1.0 | 1.0 |
| 3 | 1 | 0.0 | 0.0 |
| 3 | 4 | 0.3333333333333333 | 0.5 |
| 3 | 6 | 0.5555555555555556 | 1.0 |
+----+-----+---------------------+---------------------+--+- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
增強的聚合 Cube和Grouping 和Rollup
這幾個分析函式通常用於OLAP中，不能累加，而且需要根據不同維度上鑽和下鑽的指標統計，比如，分小時、天、月的UV數。
GROUPING SETS 在一個GROUP BY查詢中，根據不同的維度組合進行聚合，等價於將不同維度的GROUP BY結果集進行UNION ALL, 其中的GROUPING__ID，表示結果屬於哪一個分組集合。
select
user_type,
sales,
count(user_id) as pv,
GROUPING__ID
from
order_detail
group by
user_type,sales
GROUPING SETS(user_type,sales)
ORDER BY
GROUPING__ID;
+------------+--------+-----+---------------+--+
| user_type | sales | pv | grouping__id |
+------------+--------+-----+---------------+--+
| old | NULL | 3 | 1 |
| new | NULL | 7 | 1 |
| NULL | 6 | 1 | 2 |
| NULL | 5 | 2 | 2 |
| NULL | 3 | 2 | 2 |
| NULL | 2 | 2 | 2 |
| NULL | 1 | 3 | 2 |
+------------+--------+-----+---------------+--+
select
user_type,
sales,
count(user_id) as pv,
GROUPING__ID
from
order_detail
group by
user_type,sales
GROUPING SETS(user_type,sales,(user_type,sales))
ORDER BY
GROUPING__ID;
+------------+--------+-----+---------------+--+
| user_type | sales | pv | grouping__id |
+------------+--------+-----+---------------+--+
| old | NULL | 3 | 1 |
| new | NULL | 7 | 1 |
| NULL | 1 | 3 | 2 |
| NULL | 6 | 1 | 2 |
| NULL | 5 | 2 | 2 |
| NULL | 3 | 2 | 2 |
| NULL | 2 | 2 | 2 |
| old | 3 | 1 | 3 |
| old | 2 | 1 | 3 |
| old | 1 | 1 | 3 |
| new | 6 | 1 | 3 |
| new | 5 | 2 | 3 |
| new | 3 | 1 | 3 |
| new | 1 | 2 | 3 |
| new | 2 | 1 | 3 |
+------------+--------+-----+---------------+--+- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
CUBE 根據GROUP BY的維度的所有組合進行聚合。
select
user_type,
sales,
count(user_id) as pv,
GROUPING__ID
from
order_detail
group by
user_type,sales
WITH CUBE
ORDER BY
GROUPING__ID;
+------------+--------+-----+---------------+--+
| user_type | sales | pv | grouping__id |
+------------+--------+-----+---------------+--+
| NULL | NULL | 10 | 0 |
| new | NULL | 7 | 1 |
| old | NULL | 3 | 1 |
| NULL | 6 | 1 | 2 |
| NULL | 5 | 2 | 2 |
| NULL | 3 | 2 | 2 |
| NULL | 2 | 2 | 2 |
| NULL | 1 | 3 | 2 |
| old | 3 | 1 | 3 |
| old | 2 | 1 | 3 |
| old | 1 | 1 | 3 |
| new | 6 | 1 | 3 |
| new | 5 | 2 | 3 |
| new | 3 | 1 | 3 |
| new | 2 | 1 | 3 |
| new | 1 | 2 | 3 |
+------------+--------+-----+---------------+--+- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
ROLLUP 是CUBE的子集，以最左側的維度為主，從該維度進行層級聚合。
select
user_type,
sales,
count(user_id) as pv,
GROUPING__ID
from
order_detail
group by
user_type,sales
WITH ROLLUP
ORDER BY
GROUPING__ID;
+------------+--------+-----+---------------+--+
| user_type | sales | pv | grouping__id |
+------------+--------+-----+---------------+--+
| NULL | NULL | 10 | 0 |
| old | NULL | 3 | 1 |
| new | NULL | 7 | 1 |
| old | 3 | 1 | 3 |
| old | 2 | 1 | 3 |
| old | 1 | 1 | 3 |
| new | 6 | 1 | 3 |
| new | 5 | 2 | 3 |
| new | 3 | 1 | 3 |
| new | 2 | 1 | 3 |
| new | 1 | 2 | 3 |
+------------+--------+-----+---------------+--+--------------------- 本文來自 李國冬 的CSDN 部落格 ，全文地址請點選：https://blog.csdn.net/scgaliguodong123_/article/details/60135385?utm_source=copy