學習DDL語句

建立物件的語句

Create/Drop/Alter Database

Create Database

CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
  [COMMENT database_comment]
  [LOCATION hdfs_path]
  [WITH DBPROPERTIES (property_name=property_value, ...)];

Drop Database

DROP (DATABASE|SCHEMA) [IF EXISTS] database_name [RESTRICT
 
|CASCADE];

Alter Database

ALTER (DATABASE|SCHEMA) database_name SET DBPROPERTIES 
(property_name=property_value, ...); 

ALTER (DATABASE|SCHEMA) database_name SET OWNER [USER|ROLE] user_or_role;

Use Database
USE database_name;
USE DEFAULT;

Hive執行的時候，元資料儲存在關係係數據庫裡面。

Hive執行的時候需要有對映關係的資料，需要快速地讀取

Linux裡面其實有自帶的關係資料庫，但是十分不穩定，所以我們不用這個資料庫

我們自己搭建一個關係資料庫

安裝一個關係資料庫（mysql）

我們在安裝Linux的時候已經安裝了mysql

啟動mysql

檢視mysql是否已經進行監聽

3306埠，對的

連線mysql

受限我們需要驅動

設定mysql中遠端登入的問題

輸入use mysql
select * from user;

grant all on . to [email protected]’%’ identified by ‘123456’;
這個是給所有的使用者在所有的資料庫上的所有的表的所有許可權，密碼是123456

檢視一下是否成功

修改配置檔案

配置mysql路徑

修改使用者名稱和密碼

我們建立一個hive的資料庫

進入hive

啟動之後推出hive

Quite；

然後在mysql中查看錶

退出

學習hive的DDL語句

Create Table

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name    -- (Note: TEMPORARY available in Hive 0.14.0 and later)
  [(col_name data_type [COMMENT col_comment], ...)]
  [COMMENT table_comment]
  [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
  [CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS]
  [SKEWED BY (col_name, col_name, ...)                  -- (Note: Available in Hive 0.10.0 and later)]
     ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
     [STORED AS DIRECTORIES]
  [
   [ROW FORMAT row_format] 
   [STORED AS file_format]
     | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]  -- (Note: Available in Hive 0.6.0 and later)
  ]
  [LOCATION hdfs_path]
  [TBLPROPERTIES (property_name=property_value, ...)]   -- (Note: Available in Hive 0.6.0 and later)
  [AS select_statement];   -- (Note: Available in Hive 0.5.0 and later; not supported for external tables)

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
  LIKE existing_table_or_view_name
  [LOCATION hdfs_path];

data_type
  : primitive_type
  | array_type
  | map_type
  | struct_type
  | union_type  -- (Note: Available in Hive 0.7.0 and later)

primitive_type
  : TINYINT
  | SMALLINT
  | INT
  | BIGINT
  | BOOLEAN
  | FLOAT
  | DOUBLE
  | STRING
  | BINARY      -- (Note: Available in Hive 0.8.0 and later)
  | TIMESTAMP   -- (Note: Available in Hive 0.8.0 and later)
  | DECIMAL     -- (Note: Available in Hive 0.11.0 and later)
  | DECIMAL(precision, scale)  -- (Note: Available in Hive 0.13.0 and later)
  | DATE        -- (Note: Available in Hive 0.12.0 and later)
  | VARCHAR     -- (Note: Available in Hive 0.12.0 and later)
  | CHAR        -- (Note: Available in Hive 0.13.0 and later)

array_type
  : ARRAY < data_type >

map_type
  : MAP < primitive_type, data_type >

struct_type
  : STRUCT < col_name : data_type [COMMENT col_comment], ...>

union_type
   : UNIONTYPE < data_type, data_type, ... >  -- (Note: Available in Hive 0.7.0 and later)

row_format
  : DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]] [COLLECTION ITEMS TERMINATED BY char]
        [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char]
        [NULL DEFINED AS char]   -- (Note: Available in Hive 0.13 and later)
  | SERDE serde_name [WITH SERDEPROPERTIES (property_name=property_value, property_name=property_value, ...)]

file_format:
  : SEQUENCEFILE
  | TEXTFILE    -- (Default, depending on hive.default.fileformat configuration)
  | RCFILE      -- (Note: Available in Hive 0.6.0 and later)
  | ORC         -- (Note: Available in Hive 0.11.0 and later)
  | PARQUET     -- (Note: Available in Hive 0.13.0 and later)
  | AVRO        -- (Note: Available in Hive 0.14.0 and later)
  | INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname

例子：

id     int,
date   date,
name   varchar

create table table_name 
(
  id                int,
  dtDontQuery       string,
  name              string
)
partitioned by (date string)

一個例子

CREATE TABLE page_view
(
viewTime INT, 
userid BIGINT,
page_url STRING, 
referrer_url STRING,
ip STRING COMMENT 'IP Address of the User'
)
COMMENT 'This is the page view table'
PARTITIONED BY(dt STRING, country STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'  這個是分隔符，行的每一列用什麼分割
STORED AS SEQUENCEFILE;

我們建立一張表

在hive中

create table t_emp
(
id int,
name string,
age int,
dept_name string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';

我們在Linux中建立一個文字的資料檔案
Emp.txt

匯入資料

Loading files into tables
Hive does not do any transformation while loading data into tables. Load operations are currently pure copy/move operations that move datafiles into locations corresponding to Hive tables.

LOAD DATA [LOCAL] INPATH ‘filepath’ [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 …)]

Hive通過我們的環境變數找到hadoop在哪，然後連上hadoop，就會建立hive的工作目錄在hdfs上，在user下的hive下

我們查詢，在hive下面

select count(*) from t_emp;

Hive還可以使用各種集合型別

create table t_person
(
id int,
name string,
like array<string>,
tedian map<string, string>
)
row format delimited
fields terminated by ','
collection items terminated by '_'
map keys terminated by ':';

資料格式
1,,zhangsan,sports_books_TV,sex:男_color:red

載入檔案
Load data local inpath ‘root/data.exe’ into table t_person

Hive在執行的時候有一些元資料需要儲存。預設保持到DBMS。

學習DML語句

匯入資料
Loading files into tables

LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]

建立分割槽表

分割槽實際是一個資料夾，表名就是資料夾名。每個分割槽，實際上是表名這個資料夾下面的不同檔案。分割槽可以根據時間，地點等等進行分割槽，比如，每天一個分割槽，等於每天存每天的資料，或者每個城市，存放每個城市的資料。每次查詢資料的時候，只要寫下類似where pt=2010_08_23這樣的條件即可查詢指定時間的資料

Create table sxtstu(id int, sname string, city string)
Partitioned by (ds string) row format delimited fields terminated by ‘,’ stored as textfile;

我們儲存資料的時候
Load data local inpath ‘sxtstu.txt’ overwrite into table sxtstu partition(ds=’2013-07-09’);

Copying data from file:/home/Hadoop/sxtstu.txt
Copying file:file:/home/Hadoop/sxtstu.txt

Loading data to table default.sxtstu partition (ds=2013-07-09)
OK
我們嘗試建立一張表

create table dept_count(
     dname string,
     num int)
     ;


insert into table dept_count select dept_name,  count(1) from t_emp group by dept_name;

關於分割槽：

Create table dept_count
(
Num int
)
Partitioned by (dname string);

Insert into table dept_count 
partition (dname='銷售部') 
select count(1) 
from t_emp 
where dept_name='銷售部' 
group by dept_name

一些案例：

CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2))
  CLUSTERED BY (age) INTO 2 BUCKETS STORED AS ORC;

INSERT INTO TABLE students
  VALUES ('fred flintstone', 35, 1.28), ('barney rubble', 32, 2.32);


CREATE TABLE pageviews (userid VARCHAR(64), link STRING, came_from STRING)
  PARTITIONED BY (datestamp STRING) CLUSTERED BY (userid) INTO 256 BUCKETS STORED AS ORC;

INSERT INTO TABLE pageviews PARTITION (datestamp = '2014-09-23')
  VALUES ('jsmith', 'mail.com', 'sports.com'), ('jdoe', 'mail.com', null);

INSERT INTO TABLE pageviews PARTITION (datestamp)
  VALUES ('tjohnson', 'sports.com', 'finance.com', '2014-09-23'), ('tlee', 'finance.com', null, '2014-09-21');

關於import和export

EXPORT TABLE tablename [PARTITION (part_column="value"[, ...])]
  TO 'export_target_path'

IMPORT [[EXTERNAL] TABLE new_or_original_tablename [PARTITION (part_column="value"[, ...])]]
  FROM 'source_path'
  [LOCATION 'import_target_path']

匯出語句

學習資料查詢語句

類似SQL語句

create table t_stu
(
userid int,
name string,
age int,
sex int,
classid int
)
row format delimited fields terminated by ','
stored as textfile;

create table t_class
(
cid int,
name string,
teacher string
)
row format delimited fields terminated by ','
stored as textfile;

load data inpath '/pub/student.txt' into table t_stu;

1,zs,32,2,2
2,lis,23,1,2
3,ww,21,1,1

select s.*, c.name from t_stu s join t_class c on s.classid=c.cid;