oozie編譯

一、版本

• oozie版本：4.3.1
• JDK：1.8
• HADOOP：2.7.7
• hive：1.2.2
• pig：0.16.0
• sqoop：1.4.7
• spark：2.3.0
• hbase：0.94.27
• tomcat：8.0.53

二、編譯前需要修改的地方

• root pom.xml
  1. jdk版本修改

        <properties>
            <targetJavaVersion>1.8</targetJavaVersion>
            <sourceJavaVersion>1.8</sourceJavaVersion>
            <
      
     minJavaVersion>1.7</minJavaVersion>
        </properties>
     

  2. hadoop版本修改

        <properties>
            <hadoop.version>2.7.7</hadoop.version>
        </properties>
     

  3. hbase版本成新的,可能會在maven中心倉庫中找不到相應的jar包導致報錯,可不修改.
  4. hive版本修改:

        <hive.version>1.2.2</hive.version>
     

  5. sqoop 版本修改

        <
      
     sqoop.version>1.4.7</sqoop.version>
        <!--
            將sqoop版本修改為1.4.7時,還需要將<id>hadoop-2</id>下面對應的sqoop.classifier值改成hadoop260
            <sqoop.classifier>hadoop260</sqoop.classifier>
            <profile>
                 <id>hadoop-2</id>
                 <activation>
                     <activeByDefault>true</activeByDefault>
                 </activation>
                 <properties>
                     <hadoop.version>2.4.0</hadoop.version>
                     <hadoop.majorversion>2</hadoop.majorversion>
                     <pig.classifier>h2</pig.classifier>
                     <sqoop.classifier>hadoop260</sqoop.classifier>
                     <jackson.version>1.9.13</jackson.version>
                 </properties>
            </profile>
        -->
      
     
     

  6. 修改spark版本以及對應的scala版本

         <properties>
              <spark.version>2.3.0</spark.version>
              <spark.streaming.kafka.version>1.6.3</spark.streaming.kafka.version>
              <spark.bagel.version>1.6.3</spark.bagel.version>
              <spark.guava.version>14.0.1</spark.guava.version>
              <spark.scala.binary.version>2.11</spark.scala.binary.version>
         </properties>
     

  7. 修改tomcat版本(oozie預設使用的是tomcat6的)

         <tomcat.version>8.0.53</tomcat.version>
     

     同時還要修改子專案distro中的pom檔案,修改的地方如下: 將tomcat-6改成tomcat-8

        ...
        <plugin>
           ...   
          <mkdir dir="downloads"/>
          <get src="http://archive.apache.org/dist/tomcat/tomcat-8/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz"
               dest="downloads/tomcat-${tomcat.version}.tar.gz" verbose="true" skipexisting="true"/>
          ...
        </plugin>
     
     

三、編譯環境準備

• 安裝maven(可以使用最新版本) 直接去maven官網下載linux版本的maven安裝包,解壓後配置好環境變數. 同時需要在conf目錄下配置好其下載資源的localRepository,同時,如果嫌棄maven中心庫下載東西慢的話,可以使用阿里雲的mirror,但是不全.

  <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <!-- localRepository
     | The path to the local repository maven will use to store artifacts.
     |
     | Default: ${user.home}/.m2/repository
    <localRepository>/path/to/local/repo</localRepository>
    -->
  <!--配置目錄-->
  <localRepository>/usr/repo</localRepository>
  ...
  

• git安裝,可以直接使用yum -y install git 命令進行安裝.
• svn安裝,可以直接使用yum -y install svn 命令進行安裝.

四、編譯

• 進入到oozie的安裝目錄下,執行以下命令: bin/mkdistro.sh -DskipTests -Phadoop-2 -Dhadoop.auth.version=2.7.7 -Ddistcp.version=2.7.7 編譯過程中如果發現某些jar包下載失敗,需要自己下載相應的jar包放到maven的localRepository對應的目錄下. 同時,編譯過程中,可能會報如下錯誤:

   [ERROR] Java heap space -> [Help 1]
   [ERROR] 
   [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
   [ERROR] Re-run Maven using the -X switch to enable full debug logging.
   [ERROR] 
   [ERROR] For more information about the errors and possible solutions, please read the following articles:
   [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
  
  

  解決方案:在命令列輸入export MAVEN_OPTS='-Xmx512m -XX:MaxPermSize=128m'後再重新執行編譯命令.

五、編譯結果.

• 待編譯很久後,如果編譯成功,顯示如下:

      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Apache Oozie Main 4.3.1 ............................ SUCCESS [  2.022 s]
      [INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.1 hadoop-2-4.3.1 SUCCESS [  3.970 s]
      [INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.1 hadoop-2-4.3.1 SUCCESS [  0.231 s]
      [INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.1 Test hadoop-2-4.3.1 SUCCESS [  0.611 s]
      [INFO] Apache Oozie Hadoop Libs ........................... SUCCESS [  0.054 s]
      [INFO] Apache Oozie Client ................................ SUCCESS [ 22.406 s]
      [INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [  5.780 s]
      [INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [  4.434 s]
      [INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [  1.246 s]
      [INFO] Apache Oozie Core .................................. SUCCESS [01:25 min]
      [INFO] Apache Oozie Share Lib Streaming ................... SUCCESS [  8.800 s]
      [INFO] Apache Oozie Share Lib Pig ......................... SUCCESS [  7.937 s]
      [INFO] Apache Oozie Share Lib Hive ........................ SUCCESS [ 13.658 s]
      [INFO] Apache Oozie Share Lib Hive 2 ...................... SUCCESS [ 13.582 s]
      [INFO] Apache Oozie Share Lib Sqoop ....................... SUCCESS [  7.564 s]
      [INFO] Apache Oozie Examples .............................. SUCCESS [ 14.970 s]
      [INFO] Apache Oozie Share Lib Spark ....................... SUCCESS [ 31.352 s]
      [INFO] Apache Oozie Share Lib ............................. SUCCESS [ 17.920 s]
      [INFO] Apache Oozie Docs .................................. SUCCESS [ 12.683 s]
      [INFO] Apache Oozie WebApp ................................ SUCCESS [01:24 min]
      [INFO] Apache Oozie Tools ................................. SUCCESS [01:24 min]
      [INFO] Apache Oozie MiniOozie ............................. SUCCESS [01:24 min]
      [INFO] Apache Oozie Distro ................................ SUCCESS [01:24 min]
      [INFO] Apache Oozie ZooKeeper Security Tests 4.3.1 ........ SUCCESS [01:24 min]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 05:41 min
      [INFO] Finished at: 2018-09-14T23:56:36+08:00
      [INFO] ------------------------------------------------------------------------
  

  編譯成功後的安裝包在子專案distro的target目錄下的oozie-4.3.1-distro.tar.gz即是我們需要的oozie安裝包. 到此,oozie原始碼編譯完成.