1. 程式人生 > >關係資料庫資料與hadoop資料進行轉換的工具 - Sqoop

關係資料庫資料與hadoop資料進行轉換的工具 - Sqoop

Sqoop

本文所使用的Sqoop版本為1.4.6

1.官網

  http://sqoop.apache.org

2.作用

  A:可以把hadoop資料匯入到關係資料庫裡面(e.g. Hive -> Mysql)

  B:可以把關係資料庫資料匯入到hadoop裡面(e.g. Mysql -> Hive)

3.下載

  http://archive.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-1.0.0.tar.gz

4.安裝

--上傳到node1(我之前安裝的hive就在node1上面)本目錄,並且解壓
cd
tar 
-zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz --建立軟鏈 ln -sf /root/sqoop-1.4.6.bin__hadoop-2.0.4-alpha /home/sqoop-1.4.6

 

5.環境變數配置

--配置環境變數
vi /etc/profile

export HADOOP_PREFIX=$HADOOP_HOME
export PATH=$PATH:$SQOOP_HOME/bin

:wq

source /etc/profile

 

6.修改配置檔案

--修改配置檔案
cd /home/sqoop-1.4.6/conf/ cp sqoop-env-template.sh sqoop-env.sh vi sqoop-env.sh

 

7.新增驅動包

--把mysql驅動包新增到sqoop的lib目錄下面
cd
scp mysql-connector-java-5.1.23-bin.jar /home/sqoop-1.4.6/lib/

 

8.測試

sqoop version

[[email protected] ~]# sqoop version
Warning: /home/sqoop-1.4
.6/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/sqoop-1.4.6/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 19/01/04 23:15:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 Sqoop 1.4.6 git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25 Compiled by root on Mon Apr 27 14:38:36 CST 2015 sqoop list-databases -connect jdbc:mysql://node1:3306/ -username root -password '!QAZ2wsx3edc' [[email protected] ~]# sqoop list-databases -connect jdbc:mysql://node1:3306/ -username root -password '!QAZ2wsx3edc' Warning: /home/sqoop-1.4.6/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/sqoop-1.4.6/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 19/01/04 23:17:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 19/01/04 23:17:49 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 19/01/04 23:17:49 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema hive mysql performance_schema result_db spark sys

你會發現,在輸出裡面會有很多Warning

我們可以通過以下操作去掉這些Warning

--去除Warning
cd /home/sqoop-1.4.6/bin/

vi configure-sqoop

--把下面的行全部註釋掉 - 在每一行前面加 '#'
## Moved to be a runtime check in sqoop.
#if [ ! -d "${HBASE_HOME}" ]; then
#  echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
#  echo 'Please set $HBASE_HOME to the root of your HBase installation.'
#fi

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi
#if [ ! -d "${ZOOKEEPER_HOME}" ]; then
#  echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
#fi

:wq

--再次測試
[[email protected] bin]# sqoop list-databases -connect jdbc:mysql://node1:3306/ -username root -password '!QAZ2wsx3edc'
19/01/04 23:34:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 19/01/04 23:34:21 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 19/01/04 23:34:21 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema hive mysql performance_schema result_db spark sys

 

9.應用

9.1.從Mysql匯入到HDFS

準備工作:

--在資料庫裡面先建立table
CREATE TABLE t_user (id INT, name VARCHAR(20), age INT);

--插入測試資料
insert into t_user values(1, 'Tom', 20);
insert into t_user values(2, 'John', 18);
insert into t_user values(3, 'Div', 25);
insert into t_user values(4, 'Susan', 31);
insert into t_user values(5, 'Tiran', 40);
insert into t_user values(6, 'Shasita', 13);

 

查詢結果:

mysql> select * from t_user;
+------+---------+------+
| id   | name    | age  |
+------+---------+------+
|    1 | Tom     |   20 |
|    2 | John    |   18 |
|    3 | Div     |   25 |
|    4 | Susan   |   31 |
|    5 | Tiran   |   40 |
|    6 | Shasita |   13 |
+------+---------+------+
6 rows in set (0.00 sec)

 

--從mysql資料庫裡面匯出資料到Hdfs上面
sqoop import --connect jdbc:mysql://node1:3306/sqoop_db --username root --password '!QAZ2wsx3edc' --table t_user --columns id,name,age -m 1 --target-dir /sqoop_t_user


[[email protected] bin]# sqoop import --connect jdbc:mysql://node1:3306/sqoop_db --username root --password '!QAZ2wsx3edc' --table t_user --columns id,name,age -m 1 --target-dir /sqoop_t_user
19/01/04 23:54:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
19/01/04 23:54:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/01/04 23:54:30 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/01/04 23:54:30 INFO tool.CodeGenTool: Beginning code generation
19/01/04 23:54:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t_user` AS t LIMIT 1
19/01/04 23:54:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t_user` AS t LIMIT 1
19/01/04 23:54:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop-2.5
Note: /tmp/sqoop-root/compile/84e97965496cc61c73c17151375a419b/t_user.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/01/04 23:54:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/84e97965496cc61c73c17151375a419b/t_user.jar
19/01/04 23:54:33 WARN manager.MySQLManager: It looks like you are importing from mysql.
19/01/04 23:54:33 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
19/01/04 23:54:33 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
19/01/04 23:54:33 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
19/01/04 23:54:33 INFO mapreduce.ImportJobBase: Beginning import of t_user
19/01/04 23:54:33 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/01/04 23:54:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/01/04 23:54:34 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.79.138:8032
19/01/04 23:54:48 INFO db.DBInputFormat: Using read commited transaction isolation
19/01/04 23:54:48 INFO mapreduce.JobSubmitter: number of splits:1
19/01/04 23:54:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1546674829746_0001
19/01/04 23:54:49 INFO impl.YarnClientImpl: Submitted application application_1546674829746_0001
19/01/04 23:54:49 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1546674829746_0001/
19/01/04 23:54:49 INFO mapreduce.Job: Running job: job_1546674829746_0001
19/01/04 23:54:59 INFO mapreduce.Job: Job job_1546674829746_0001 running in uber mode : false
19/01/04 23:54:59 INFO mapreduce.Job:  map 0% reduce 0%
19/01/04 23:55:06 INFO mapreduce.Job:  map 100% reduce 0%
19/01/04 23:55:06 INFO mapreduce.Job: Job job_1546674829746_0001 completed successfully
19/01/04 23:55:06 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=116299
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=87
        HDFS: Number of bytes written=63
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=4153
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=4153
        Total vcore-seconds taken by all map tasks=4153
        Total megabyte-seconds taken by all map tasks=4252672
    Map-Reduce Framework
        Map input records=6
        Map output records=6
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=69
        CPU time spent (ms)=1170
        Physical memory (bytes) snapshot=175808512
        Virtual memory (bytes) snapshot=893071360
        Total committed heap usage (bytes)=84934656
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=63
19/01/04 23:55:06 INFO mapreduce.ImportJobBase: Transferred 63 bytes in 32.3608 seconds (1.9468 bytes/sec)
19/01/04 23:55:06 INFO mapreduce.ImportJobBase: Retrieved 6 records.

 

執行效果:

 

官網提供另一種方式,即讀取檔案的方式來實現上面的匯入功能

cd 
mkdir mysqoopdir
cd mysqoopdir

vi mysql_to_hdfs


import 
--connect 
jdbc:mysql://node1:3306/sqoop_db 
--username 
root 
--password 
'!QAZ2wsx3edc' 
--table 
t_user 
--columns 
id,name,age 
-m 
1 
--target-dir 
/sqoop_t_user
--delete-target-dir

:wq

sqoop --options-file mysql_to_hdfs

 

我們可以通過Hive,來驗證匯入結果

[[email protected] bin]# ./hive
19/01/05 00:03:29 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead

Logging initialized using configuration in jar:file:/root/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
hive> dfs -cat /sqoop_t_user/*;
1,Tom,20
2,John,18
3,Div,25
4,Susan,31
5,Tiran,40
6,Shasita,13

 

我們看到的結果和mysql裡面的資料一樣。

應用場景:如果現在我們的需要處理/分析的資料都存在Mysql資料庫裡面,並且資料量比較大,我們想要通過離線分析這些資料。這時,我們就可以把Mysql裡面的資料通過Sqoop匯入到Hdfs裡面,進行分析處理。

 

匯入查詢結果:

--匯入查詢結果
cd mysqoopdir

vi mysql_query_to_hdfs


import
--connect
jdbc:mysql://node1:3306/sqoop_db
--username
root
--password
'!QAZ2wsx3edc'
-e 
select id, name from t_user where id >= 1 and $CONDITIONS
-m
1
--target-dir
/sqoop_t_user
--delete-target-dir

:wq

sqoop --options-file mysql_query_to_hdfs

--檢驗
hive> dfs -cat /sqoop_t_user/*;
1,Tom
2,John
3,Div
4,Susan
5,Tiran
6,Shasita

 

9.2Mysql匯入資料到Hive

--mysql匯入到Hive
cd mysqoopdir

vi mysql_to_hive


import
--connect
jdbc:mysql://node1:3306/sqoop_db
--username
root
--password
'!QAZ2wsx3edc'
--table
t_user
-m
1
--create-hive-table
--target-dir
/sqoop_mysql_to_hive/
--hive-home
/home/hive/
--hive-import
--hive-table
t_sqoop_mysql_t_user_to_hive
--create-hive-table


:wq

sqoop --options-file mysql_to_hive

--檢驗
hive> select * from t_sqoop_mysql_t_user_to_hive;
OK
1    Tom    20
2    John    18
3    Div    25
4    Susan    31
5    Tiran    40
6    Shasita    13
Time taken: 0.577 seconds, Fetched: 6 row(s)

 

9.3.Mysql匯入到Hbase

--mysql匯入到Hbase
cd mysqoopdir

vi mysql_to_hbase


import
--connect
jdbc:mysql://node1:3306/sqoop_db
--username
root
--password
'!QAZ2wsx3edc'
--table
t_user
-m
1
--hbase-table
t_sqoop_mysql_t_user_to_hbase
--hbase-row-key
id
--hbase-create-table
--column-family
cf


:wq

sqoop --options-file mysql_to_hbase

--檢驗
hbase(main):004:0> scan 't_sqoop_mysql_t_user_to_hbase'
ROW                     COLUMN+CELL                                                        
 1                      column=cf:age, timestamp=1546680684317, value=20                   
 1                      column=cf:name, timestamp=1546680684317, value=Tom                 
 2                      column=cf:age, timestamp=1546680684317, value=18                   
 2                      column=cf:name, timestamp=1546680684317, value=John                
 3                      column=cf:age, timestamp=1546680684317, value=25                   
 3                      column=cf:name, timestamp=1546680684317, value=Div                 
 4                      column=cf:age, timestamp=1546680684317, value=31                   
 4                      column=cf:name, timestamp=1546680684317, value=Susan               
 5                      column=cf:age, timestamp=1546680684317, value=40                   
 5                      column=cf:name, timestamp=1546680684317, value=Tiran               
 6                      column=cf:age, timestamp=1546680684317, value=13                   
 6                      column=cf:name, timestamp=1546680684317, value=Shasita             
6 row(s) in 0.1730 seconds

 

9.4.Hdfs匯出到Mysql

--Hdfs匯出到Mysql
cd mysqoopdir

vi hdfs_to_mysql


export
--connect
jdbc:mysql://node1:3306/sqoop_db
--username
root
--password
'!QAZ2wsx3edc'
--table
t_hdfs_user_to_mysql
-m
1
--columns
id,name,age
--export-dir
/sqoop_t_user/


:wq


--進入node1
mysql -u root -p
!QAZ2wsx3edc
show database;
use sqoop_db;
show tables;
--建立t_hdfs_user_to_mysql表
CREATE TABLE t_hdfs_user_to_mysql (id INT, name VARCHAR(20), age INT);


cd mysqoopdir
sqoop --options-file hdfs_to_mysql

--檢驗
--執行匯入前
mysql> select * from t_hdfs_user_to_mysql;
Empty set (0.00 sec)

--執行匯入後
mysql> mysql> select * from t_hdfs_user_to_mysql;
+------+---------+------+
| id   | name    | age  |
+------+---------+------+
|    1 | Tom     |   20 |
|    2 | John    |   18 |
|    3 | Div     |   25 |
|    4 | Susan   |   31 |
|    5 | Tiran   |   40 |
|    6 | Shasita |   13 |
+------+---------+------+
6 rows in set (0.00 sec)

 

========================================================

More reading,and english is important.

I'm Hongten

 

大哥哥大姐姐,覺得有用打賞點哦!你的支援是我最大的動力。謝謝。
Hongten部落格排名在100名以內。粉絲過千。
Hongten出品,必是精品。

E | [email protected]  B | http://www.cnblogs.com/hongten

========================================================