周海漢/文 2013.8.20 http://abloz.com

摘要：

1. sqoop 1.99的安裝配置
2. client使用
3. 從HBase，Hive導資料到mysql

版本 sqoop-1.99.2-bin-hadoop100

Sqoop是Hadoop系統資料和RDBMS互導資料的工具。1.99新版的一個包裡包含兩個部分：客戶端和伺服器端。必須在叢集中安裝好一個節點的伺服器端。所有客戶端和該伺服器端相連。伺服器端作為mapreduce的client，所以Hadoop必須和Sqoop伺服器端裝在一起。客戶端則不限。這種設計讓導資料更靈活。原1.44版則沒有區分伺服器端和客戶端。

伺服器端安裝

1.確認本機有hadoop hadoop fs -ls

由於hadoop的主版本1.xx和2.xx不相容，所以sqoop的二進位制版本也是分100和200的。如我在hadoop 1.1.2中使用sqoop-1.99.2-bin-hadoop100來進行配套。 解壓 tar zxvf sqoop-1.99.2-bin-hadoop100.tar.gz cd sqoop-1.99.2-bin-hadoop100

1. 安裝依賴庫和元件

./bin/addtowar.sh -hadoop-auto

[[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto

Non of expected directories with Hadoop exists

Usage : addtowar.sh Options: -hadoop-auto Try to guess hadoop version and path -hadoop-version HADOOP_VERSION Specify used version -hadoop-path HADOOP_PATHS Where to find hadoop jars (multiple paths with Hadoop jars separated by ‘:’) -jars JARS_PATH Special jars that should be added (multiple JAR paths separated by ‘:’) -war SQOOP_WAR Target Sqoop war file where all jars should be ingested

由於我的hadoop沒有安裝到系統路徑，所以需修改配置檔案。 [[email protected] sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh hadoopPossiblePaths=”/home/hadoop/hadoop-1.1.2 /usr/lib/hadoop /usr/lib/hadoop-mapreduce/ /usr/lib/hadoop-yarn/ /usr/lib/hadoop-hdfs”

[[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto ./bin/addtowar.sh: line 126: [: missing `]’ Hadoop version: 1.1.2 Hadoop path: /home/hadoop/hadoop-1.1.2 Extra jars:

Injecting following Hadoop JARs

/home/hadoop/hadoop-1.1.2/hadoop-core-1.1.2.jar /home/hadoop/hadoop-1.1.2/lib/jackson-core-asl-1.8.8.jar /home/hadoop/hadoop-1.1.2/lib/jackson-mapper-asl-1.8.8.jar /home/hadoop/hadoop-1.1.2/lib/commons-configuration-1.6.jar /home/hadoop/hadoop-1.1.2/lib/commons-logging-api-1.0.4.jar /home/hadoop/hadoop-1.1.2/lib/slf4j-api-1.4.3.jar /home/hadoop/hadoop-1.1.2/lib/slf4j-log4j12-1.4.3.jar

Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:36:01.263437795

New Sqoop WAR file with added ‘Hadoop JARs’ at ./bin/../server/webapps/sqoop.war

指令碼126行]有個空格問題，但不影響結果。改後： [[email protected] sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh [[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto Hadoop version: 1.1.2 Hadoop path: /home/hadoop/hadoop-1.1.2 Extra jars:

Specified Sqoop WAR ‘./bin/../server/webapps/sqoop.war’ already contains Hadoop JAR files

也可以指定,-hadoop-version指定版本，-hadoop-path 指定目錄。如果各部分安裝在多個目錄，則用:分隔。 如： hadoop 1.0 ./bin/addtowar.sh -hadoop-version 1.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-mpred

hadoop 2.0 ./bin/addtowar.sh -hadoop-version 2.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-yarn

也可以用addtowar.sh的-jars引數來繫結其他的jar檔案。 繫結jdbc mysql庫： 由於不同協議，sqoop沒有自帶mysql jdbc庫，需下載，並繫結： http://dev.mysql.com/downloads/mirror.php?id=13597 wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.0.8.tar.gz/from/http://cdn.mysql.com/

注意下載的是tar.gz, 裡面有原始碼和jar包。只需解壓後使用jar包即可。

[[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -jars /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar Hadoop version: Hadoop path: Extra jars: /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar

Injecting following additional JARs

/home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar

Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:49:32.401896012

New Sqoop WAR file with added ‘JARs’ at ./bin/../server/webapps/sqoop.war

3.配置sqoop伺服器 server/conf 裡面存放伺服器的配置檔案。包括tomcat等配置。但預設配置PropertiesConfigurationProvider足夠用。如需修改，編輯sqoop_bootstrap.properties的sqoop.config.provider即可。 [[email protected] sqoop-1.99.2-bin-hadoop100]$ cd server/conf [[email protected] conf]$ ls catalina.policy catalina.properties context.xml logging.properties server.xml sqoop_bootstrap.properties sqoop.properties tomcat-users.xml web.xml [[email protected] conf]$ cat sqoop_bootstrap.properties sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider

sqoop.properties包含其他的一些修改。可能需要微調，以適應環境需要。 #org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/ org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-1.1.2/conf/

1. 啟動和停止伺服器 ./bin/sqoop.sh server start

[[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server start Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp Using JRE_HOME: /usr/java/jdk1.6.0_45 Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar

[[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server stop Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp Using JRE_HOME: /usr/java/jdk1.6.0_45 Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar

5.客戶端安裝 客戶端無需配置，只需將下載版本解壓即可。 [[email protected] sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh client Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100… Aug 20, 2013 10:07:54 AM java.util.prefs.FileSystemPreferences$2 run INFO: Created user preferences directory. Sqoop Shell: Type ‘help’ or ‘h’ for help.

sqoop:000> 或執行sqoop指令碼： sqoop.sh client /path/to/your/script.sqoop

具體使用在下一篇詳述。 6.參考： http://sqoop.apache.org/docs/1.99.2/Installation.html http://sqoop.apache.org/docs/1.99.2/Sqoop5MinutesDemo.html http://sqoop.apache.org/docs/1.99.2/CommandLineClient.html

如非註明轉載, 均為原創. 本站遵循知識共享CC協議,轉載請註明來源