Scoop是用來實現HDFS文件系統和關系型數據庫如MySQL之間數據傳輸和轉換的工具。

從MySQL導出到HDFS可以通過--table, --columns and --where等設置數據抽出的條件。但是同時也只是自由sql語句（Free-form Query ）的方式抽出數據。此時我們用--query加sql語句方式自由抽取數據。

1，必須制定目標文件的位置--target-dir

2，必須使用$CONDITIONS關鍵字，

3，你也可以選擇使用--split-by分片(分區，結果分成多個小文件，請參考mapreduce分區)

我們主要討論$CONDITIONS關鍵字的作用是什麽。

1如果直接輸出，這裏面是空的條件

2，我們在執行log中發現被替換成了1=0

sqoop import   --connect jdbc:mysql://server74:3306/Server74   --username root  --password 123456  --target-dir /sqoopout2  --m 1 --delete-target-dir 
 --query ‘select id,name,deg from emp where id>1202 and $CONDITIONS‘ 

[root@server72 sqoop]# sqoop import   --connect jdbc:mysql://server74:3306/Server74   --username root  --password 123456  --target-dir /sqoopout2  
--m 1 --delete-target-dir  --query ‘select id,name,deg from emp where id>1202 and $CONDITIONS‘ 
Warning: /usr/local/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/11/10 13:42:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/11/10 13:42:14 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/11/10 13:42:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/11/10 13:42:16 INFO tool.CodeGenTool: Beginning code generation
17/11/10 13:42:18 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0)
 
 
17/11/10 13:42:18 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0) 
17/11/10 13:42:18 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0) 
17/11/10 13:42:18 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: /tmp/sqoop-root/compile/ac7745794cf5f0bf5859e7e8369a8c5f/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/11/10 13:42:31 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ac7745794cf5f0bf5859e7e8369a8c5f/QueryResult.jar
17/11/10 13:42:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/10 13:42:41 INFO tool.ImportTool: Destination directory /sqoopout2 deleted.
17/11/10 13:42:41 INFO mapreduce.ImportJobBase: Beginning query import.
17/11/10 13:42:41 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/11/10 13:42:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/11/10 13:42:43 INFO client.RMProxy: Connecting to ResourceManager at server71/192.168.32.71:8032
17/11/10 13:42:58 INFO db.DBInputFormat: Using read commited transaction isolation
17/11/10 13:42:58 INFO mapreduce.JobSubmitter: number of splits:1
17/11/10 13:43:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1510279795921_0011
17/11/10 13:43:03 INFO impl.YarnClientImpl: Submitted application application_1510279795921_0011
17/11/10 13:43:04 INFO mapreduce.Job: The url to track the job: http://server71:8088/proxy/application_1510279795921_0011/
17/11/10 13:43:04 INFO mapreduce.Job: Running job: job_1510279795921_0011
17/11/10 13:44:01 INFO mapreduce.Job: Job job_1510279795921_0011 running in uber mode : false
17/11/10 13:44:01 INFO mapreduce.Job:  map 0% reduce 0%
17/11/10 13:44:58 INFO mapreduce.Job:  map 100% reduce 0%
17/11/10 13:45:00 INFO mapreduce.Job: Job job_1510279795921_0011 completed successfully
17/11/10 13:45:01 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=124473
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=87
        HDFS: Number of bytes written=61
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=45099
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=45099
        Total vcore-milliseconds taken by all map tasks=45099
        Total megabyte-milliseconds taken by all map tasks=46181376
    Map-Reduce Framework
        Map input records=3
        Map output records=3
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=370
        CPU time spent (ms)=6380
        Physical memory (bytes) snapshot=106733568
        Virtual memory (bytes) snapshot=842854400
        Total committed heap usage (bytes)=16982016
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=61
17/11/10 13:45:01 INFO mapreduce.ImportJobBase: Transferred 61 bytes in 139.3429 seconds (0.4378 bytes/sec)
17/11/10 13:45:01 INFO mapreduce.ImportJobBase: Retrieved 3 records.

輸出結果查看，發現1202以上的數據被正常抽出
 

[root@server72 sqoop]# hdfs dfs -cat /sqoopout2/part-m-00000
17/11/10 13:48:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1203,khalil,php dev
1204,prasanth,php dev
1205,kranthi,admin 

通過以上過程，我們得知一點：$CONTITONS是linux系統的變量，在執行過程中被賦值為（1=0），雖然實際執行的這個sql很奇怪。

現在正式開始研究CONTITONS到底是什麽，所以我們先查看官方文檔。

If you want to import the results of a query in parallel, then each map task will need to execute a copy of the query, with results partitioned by bounding conditions inferred by Sqoop. Your query must include the token $CONDITIONS which each Sqoop process will replace with a unique condition expression. You must also select a splitting column with --split-by.

如果你想通過並行的方式導入結果，每個map task需要執行sql查詢語句的副本，結果會根據sqoop推測的邊界條件分區。query必須包含$CONDITIONS。這樣每個scoop程序都會被替換為一個獨立的條件。同時你必須指定--split-by.分區

For example:

$ sqoop import   --query ‘SELECT a.*, b.* FROM a JOIN b on (a.id == b.id) WHERE $CONDITIONS‘   --split-by a.id --target-dir /user/foo/joinresults

直接理解可能有點困難，我先修改一些條件，大家觀察joblog的區別。

sqoop import --connect jdbc:mysql://server74:3306/Server74 --username root --password 123456 --target-dir /sqoopout2

　　　　　　--m 2 --delete-target-dir --query ‘select id,name,deg from emp where id>1202 and $CONDITIONS‘

　　　　　　--split-by id

我按照要求添加了--split-by id 分區，並設置map task數量為2

[root@server72 sqoop]# sqoop import   --connect jdbc:mysql://server74:3306/Server74   --username root 
 --password 123456  --target-dir /sqoopout2  --m 2 --delete-target-dir  --query ‘select id,name,deg from emp where id>1202 and $CONDITIONS‘ --split-by id 
Warning: /usr/local/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/11/10 13:50:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/11/10 13:50:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/11/10 13:50:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/11/10 13:50:28 INFO tool.CodeGenTool: Beginning code generation
17/11/10 13:50:30 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0) 
17/11/10 13:50:31 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0) 
17/11/10 13:50:31 INFO manager.SqlManager: Executing SQL statement: select id,name,deg from emp where id>1202 and  (1 = 0) 
17/11/10 13:50:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: /tmp/sqoop-root/compile/1024341fa58082466565e5bd648cb10e/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/11/10 13:50:43 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/1024341fa58082466565e5bd648cb10e/QueryResult.jar
17/11/10 13:50:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/10 13:50:55 INFO tool.ImportTool: Destination directory /sqoopout2 deleted.
17/11/10 13:50:55 INFO mapreduce.ImportJobBase: Beginning query import.
17/11/10 13:50:55 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/11/10 13:50:55 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/11/10 13:50:56 INFO client.RMProxy: Connecting to ResourceManager at server71/192.168.32.71:8032
17/11/10 13:51:12 INFO db.DBInputFormat: Using read commited transaction isolation
17/11/10 13:51:12 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(id), MAX(id) FROM (select id,name,deg from emp where id>1202 and  (1 = 1) ) AS t1
17/11/10 13:51:12 INFO mapreduce.JobSubmitter: number of splits:3
17/11/10 13:51:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1510279795921_0012
17/11/10 13:51:18 INFO impl.YarnClientImpl: Submitted application application_1510279795921_0012
17/11/10 13:51:19 INFO mapreduce.Job: The url to track the job: http://server71:8088/proxy/application_1510279795921_0012/
17/11/10 13:51:19 INFO mapreduce.Job: Running job: job_1510279795921_0012
17/11/10 13:52:19 INFO mapreduce.Job: Job job_1510279795921_0012 running in uber mode : false
17/11/10 13:52:19 INFO mapreduce.Job:  map 0% reduce 0%
17/11/10 13:53:23 INFO mapreduce.Job:  map 33% reduce 0%
17/11/10 13:54:19 INFO mapreduce.Job:  map 67% reduce 0%
17/11/10 13:54:20 INFO mapreduce.Job:  map 100% reduce 0%
17/11/10 13:54:24 INFO mapreduce.Job: Job job_1510279795921_0012 completed successfully
17/11/10 13:54:25 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=374526
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=301
        HDFS: Number of bytes written=61
        HDFS: Number of read operations=12
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=6
    Job Counters 
        Killed map tasks=2
        Launched map tasks=5
        Other local map tasks=5
        Total time spent by all maps in occupied slots (ms)=349539
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=349539
        Total vcore-milliseconds taken by all map tasks=349539
        Total megabyte-milliseconds taken by all map tasks=357927936
    Map-Reduce Framework
        Map input records=3
        Map output records=3
        Input split bytes=301
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=3013
        CPU time spent (ms)=21550
        Physical memory (bytes) snapshot=321351680
        Virtual memory (bytes) snapshot=2528706560
        Total committed heap usage (bytes)=52994048
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=61

Sqoop--Free-form Query Imports 自由查詢模式下$CONDITIONS關鍵字的作用