Sqoop簡介及安裝部署
阿新 • • 發佈：2018-07-03
cdh5 ive not 簡介 ins one generate and lin 簡介：
Apache Sqoop是專為Apache Hadoop和結構化數據存儲如關系數據庫之間的數據轉換工具的有效工具。你可以使用Sqoop從外部結構化數據存儲的數據導入到Hadoop分布式文件系統或相關系統如Hive和HBase。相反，Sqoop可以用來從Hadoop的數據提取和導出到外部結構化數據存儲如關系數據庫和企業數據倉庫。
Sqoop專為大數據批量傳輸設計，能夠分割數據集並創建Hadoop任務來處理每個區塊。
下面介紹下安裝部署的步驟：
1.下載安裝包並解壓
下載地址 作者使用的是sqoop-1.4.6-cdh5.7.0.tar.gz版本
# wget下載安裝包 也可下載到本地 然後上傳至Linux [hadoop@hadoop000 software]$ pwd /home/hadoop/software [hadoop@hadoop000 software]$ wget http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.7.0.tar.gz # 解壓 [hadoop@hadoop000 software]$ tar -xzvf sqoop-1.4.6-cdh5.7.0.tar.gz -C /home/hadoop/app/ # 查看解壓目錄 [hadoop@hadoop000 sqoop-1.4.6-cdh5.7.0]$ ls -lh total 1.9M drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 bin --執行腳本目錄 -rw-rw-r-- 1 hadoop hadoop 60K Mar 24 2016 build.xml -rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 cdh.build.properties -rw-rw-r-- 1 hadoop hadoop 35K Mar 24 2016 CHANGELOG.txt drwxr-xr-x 4 hadoop hadoop 4.0K Jul 3 16:00 cloudera -rw-rw-r-- 1 hadoop hadoop 6.8K Mar 24 2016 cloudera-pom.xml -rw-rw-r-- 1 hadoop hadoop 9.7K Mar 24 2016 COMPILING.txt drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 conf --配置文件目錄 drwxr-xr-x 5 hadoop hadoop 4.0K Jul 3 16:00 docs --文檔 drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 ivy -rw-rw-r-- 1 hadoop hadoop 17K Mar 24 2016 ivy.xml drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 lib --lib依賴包 -rw-rw-r-- 1 hadoop hadoop 15K Mar 24 2016 LICENSE.txt -rw-rw-r-- 1 hadoop hadoop 505 Mar 24 2016 NOTICE.txt -rw-rw-r-- 1 hadoop hadoop 19K Mar 24 2016 pom-old.xml -rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 README.txt -rw-rw-r-- 1 hadoop hadoop 1012K Mar 24 2016 sqoop-1.4.6-cdh5.7.0.jar --完整jar包 -rw-rw-r-- 1 hadoop hadoop 6.5K Mar 24 2016 sqoop-patch-review.py -rw-rw-r-- 1 hadoop hadoop 641K Mar 24 2016 sqoop-test-1.4.6-cdh5.7.0.jar drwxr-xr-x 7 hadoop hadoop 4.0K Mar 24 2016 src --源碼 drwxr-xr-x 4 hadoop hadoop 4.0K Jul 3 16:00 testdata
2.配置環境變量
# 添加sqoop環境變量 可加入全局 也可只配置個人環境變量
[hadoop@hadoop000 ~]$ sudo vi/etc/profile
export SQOOP_HOME=/home/hadoop/app/sqoop-1.4.6-cdh5.7.0
export PATH=$SQOOP_HOME/bin:$PATH
[hadoop@hadoop000 ~]$ source /etc/profile3.修改sqoop配置文件
[hadoop@hadoop000 conf]$ pwd /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/conf [hadoop@hadoop000 conf]$ cp sqoop-env-template.sh sqoop-env.sh # 添加hadoop及hive目錄 [hadoop@hadoop000 conf]$ vi sqoop-env.sh #Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0 #Set path to where hadoop-*-core.jar is available export HADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0 #set the path to where bin/hbase is available #export HBASE_HOME= #Set the path to where bin/hive is available export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0 #Set the path for where zookeper config dir is #export ZOOCFGDIR=
4.拷貝jdbc驅動包到sqoop/lib目錄下
# 將Hive lib目錄下的mysql驅動包拷貝過來
[hadoop@hadoop000 lib]$ pwd
/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/lib
[hadoop@hadoop000 lib]$ cp /home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/mysql-connector-java-5.1.46.jar .5.sqoop簡單測試使用
# 查看命令幫助 [hadoop@hadoop000 ~]$ sqoop help Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 18/07/03 16:23:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0 usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS import-mainframe Import datasets from a mainframe server to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information See ‘sqoop help COMMAND‘ for information on a specific command. # 查看sqoop版本 [hadoop@hadoop000 ~]$ sqoop version Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 18/07/03 16:23:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0 Sqoop 1.4.6-cdh5.7.0 git commit id Compiled by jenkins on Wed Mar 23 11:30:51 PDT 2016 # 這裏的警告是因為我沒有配置hbase，zookeeper，HCatalog
Sqoop簡介及安裝部署