Windows 10 上編譯 Hadoop
阿新 • • 發佈：2019-04-26
rem snap mpi sum isp 成功 event pip text 
下載源碼
源碼下載地址（Source download）：https://hadoop.apache.org/releases.html
這裏以 2.9.2 為例，查看源碼中的編譯說明文件 BUILDING.txt，截取 windows 部分
Requirements: * Windows System * JDK 1.7 or 1.8 * Maven 3.0 or later * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer * Windows SDK 7.1 or Visual Studio 2010 Professional * Windows SDK 8.1 (if building CPU rate control for the container executor) * zlib headers (if building native code bindings for zlib) * Internet connection for first build (to fetch all Maven and Hadoop dependencies) * Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH. * Python ( for generation of docs using ‘mvn site‘) Unix command-line tools are also included with the Windows Git package which can be downloaded from http://git-scm.com/downloads If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express. It does not support compiling for 64-bit, which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported.
環境準備
1.JDK
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
# 設置系統環境變量 setx /M JAVA_HOME "D:\hadoop\jdk1.8.0_192" setx /M Path "%Path%;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin"
2.Maven
http://maven.apache.org/download.cgi
# 設置系統環境變量 setx /M M2_HOME "D:\hadoop\apache-maven-3.6.1" setx /M Path "%Path%;%M2_HOME%\bin"
conf\settings.xml 倉庫配置
<!-- 本地倉庫路徑 --> <localRepository>D:\hadoop\repo</localRepository> <!-- 網絡倉庫地址 --> <mirrors> <mirror> <id>central</id> <mirrorOf>central</mirrorOf> <name>aliyunmaven</name> <url>https://maven.aliyun.com/repository/central</url> </mirror> <mirror> <id>apache.snapshots.https</id> <mirrorOf>apache.snapshots.https</mirrorOf> <name>aliyunmaven</name> <url>https://maven.aliyun.com/repository/apache-snapshots</url> </mirror> </mirrors>
3.ProtocolBuffer
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
註：除了下載 protobuf 源碼外，還需要下載相應版本的編譯過的用於 Windows 平臺的 protoc 命令（protoc-2.5.0-win32.zip）,該命令用於將 .proto 文件轉化為 Java 或 C++ 源文件。
將兩個壓縮包解壓，然後將 protoc.exe 復制到 protobuf-2.5.0\src 目錄下
# 安裝 ProtocolBuffer cd D:\hadoop\protobuf-2.5.0\java mvn test mvn install # 設置系統環境變量 setx /M Path "%Path%;D:\hadoop\protobuf-2.5.0\src" # 查看版本 protoc --version
4.CMake
https://cmake.org/download/
# 設置系統環境變量 setx /M Path "%Path%;D:\hadoop\cmake-3.14.3-win64-x64\bin"
5.Visual Studio 2010 Professional
文件名 cn_visual_studio_2010_professional_x86_dvd_532145.iso SHA1 33D323446131AB9565082D65C9C380BBD7FF228F 文件大小 2.41GB 發布時間 2010-05-26
ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/
6.Zlib
http://www.zlib.net/
使用 MSVC 方式編譯，打開開始菜單，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理員身份運行
cd D:\hadoop\zlib-1.2.11 nmake -f win32/Makefile.msc # 編譯完成後在該目錄下可看到 zlib1.dll 文件
設置系統環境變量
setx /M ZLIB_HOME "D:\hadoop\zlib-1.2.11"
7.Git（需要 bash 命令）
https://git-scm.com/download/
# 設置系統環境變量 setx /M Path "%Path%;D:\hadoop\PortableGit\bin"
編譯
1.設置Platform
setx /M Platform "x64"
2.解壓源碼，開始編譯
打開開始菜單，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理員身份運行
cd D:\hadoop\hadoop-2.9.2-src mvn package -Pdist,native-win -DskipTests -Dtar # 編譯完成後，編譯好的文件在目錄 hadoop-2.9.2-src\hadoop-dist\target\ 中
編譯失敗時多編譯幾次，或自己更換下 Maven 源再試，一些依賴的下載不是很穩定。附上編譯成功的日誌
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.205 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min] [INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.259 s] [INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.421 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.064 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min] [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min] [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.279 s] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.839 s] [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.044 s] [INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.044 s] [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.375 s] [INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min] [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.536 s] [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.050 s] [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s] [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s] [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.167 s] [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min] [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s] [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s] [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.589 s] [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.674 s] [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.209 s] [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.911 s] [INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.163 s] [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min] [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min] [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.045 s] [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.792 s] [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.213 s] [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.042 s] [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.045 s] [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 5.462 s] [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.134 s] [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s] [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s] [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.323 s] [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.160 s] [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.593 s] [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 2.615 s] [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.280 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.254 s] [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.127 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 5.503 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.260 s] [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.315 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.355 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.324 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.477 s] [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.219 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.615 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.040 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.513 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s] [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min] [INFO] Apache Hadoop Client ............................... SUCCESS [ 5.923 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.596 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.338 s] [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min] [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.047 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s] [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.554 s] [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.049 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 55:18 min [INFO] Finished at: 2019-04-25T22:47:26+08:00 [INFO] ------------------------------------------------------------------------View Code
Windows 10編譯Hadoop 2.6.0源碼 - 作業部落 Cmd Markdown 編輯閱讀器 （快照）
https://www.cnblogs.com/guoxiaoqian/p/4328812.html
Windows 10 上編譯 Hadoop