sqoop從oracle資料庫抽取資料,匯入到hive
環境:
hadoop-2.7.5
sqoop-1.4.7
zookeeper-3.4.10
hive-2.3.3 (使用mysql配置元資料庫)
jdk1.8.0_151
oracle 11.2.0.3.0
經過一番baidu,總算初步成功,現在記錄一下中間過程.
1.拷貝hive/conf/hive_site.xml到sqoop/conf目錄
2.配置sqoop-evn.sh,將變數設定為對應的目錄.
export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.5
export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.5
export HIVE_HOME=/home/hadoop/hive-2.3.3
export ZOOCFGDIR=/home/hadoop/zookeeper-3.4.10
3.拷貝hive/lib/derby-10.10.2.0.jar到sqoop/lib
4.拷貝/home/hadoop/hadoop-2.7.5/share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar到sqoop/lib
5.拷貝hive/lib/jackson-databind-2.6.5.jar, jackson-core-2.6.5.jar, jackson-annotations-2.6.0.jar到sqoop/lib
6.拷貝oracle11g ojdbc6.jar到sqoop/lib(從linux環境的oracle目錄中拷貝)
7.修改/home/hadoop/jdk1.8.0_151/jre/lib/security/java.policy,增加 permission javax.management.MBeanTrustPermission "register";
8.修改/home/~/.bashrc,在最後增加 export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
執行
sqoop import --connect jdbc:oracle:thin:@10.0.3.3:1521:sid --username abc --password r123 --table CMX.SALES -m 1 --hive-import --hive-overwrite --hive-database ra --hive-table "cmx_sales" --null-non-string '' --null-string '' --delete-target-dir --hive-drop-import-delims
該命令能夠抽取oracle中cmx_sales表的資料到hive中,現在面臨問題,hive中自動建立的表,欄位型別與oracle的不一樣,造成資料精度不一樣.
比如oracle中欄位為number(10),到hive中變為double,資料也從1000變成1000.0.
續待...