1. 程式人生 > >java 解決Could not locate executable null\bin\winutils.exe in the Hadoop bin

java 解決Could not locate executable null\bin\winutils.exe in the Hadoop bin

問題描述
在windows環境下執行連線hadoop服務的程式報告以下錯誤資訊:

og4j:ERROR Could not find value for key log4j.appender.logRollingFile
log4j:ERROR Could not instantiate appender named "logRollingFile".
[ERROR] 2018-09-17 17:40:02,151 method:org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:374)
Failed to locate the winutils binary in
the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) at org.apache.hadoop.util.Shell.<clinit>(Shell.java
:364) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java
:2668) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160) at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157) at com.yue.modular.hdfsClient.main(hdfsClient.java:31) [WARN ] 2018-09-17 17:40:03,432 method:org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62) Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

原因分析
根據錯誤資訊進入hadoop原始碼發現:

static {
        WINDOWS = osType == Shell.OSType.OS_TYPE_WIN;
        SOLARIS = osType == Shell.OSType.OS_TYPE_SOLARIS;
        MAC = osType == Shell.OSType.OS_TYPE_MAC;
        FREEBSD = osType == Shell.OSType.OS_TYPE_FREEBSD;
        LINUX = osType == Shell.OSType.OS_TYPE_LINUX;
        OTHER = osType == Shell.OSType.OS_TYPE_OTHER;
        PPC_64 = System.getProperties().getProperty("os.arch").contains("ppc64");

        //注意這行
        HADOOP_HOME_DIR = checkHadoopHome();
        WINUTILS = getWinUtilsPath();
        isSetsidAvailable = isSetsidSupported();
        TOKEN_SEPARATOR_REGEX = WINDOWS ? "[|\n\r]" : "[ \t\n\r\f]";
    }

private static String checkHadoopHome() {

        //高能高能~~~~~~~~~
        String home = System.getProperty("hadoop.home.dir");
        if (home == null) {
            home = System.getenv("HADOOP_HOME");
        }

        try {
            if (home == null) {
                throw new IOException("HADOOP_HOME or hadoop.home.dir are not set.");
            }

            if (home.startsWith("\"") && home.endsWith("\"")) {
                home = home.substring(1, home.length() - 1);
            }

            File homedir = new File(home);
            if (!homedir.isAbsolute() || !homedir.exists() || !homedir.isDirectory()) {
                throw new IOException("Hadoop home directory " + homedir + " does not exist, is not a directory, or is not an absolute path.");
            }

            home = homedir.getCanonicalPath();
        } catch (IOException var2) {
            if (LOG.isDebugEnabled()) {
                LOG.debug("Failed to detect a valid hadoop home directory", var2);
            }

            home = null;
        }

        return home;
    }

public static final String getQualifiedBinPath(String executable) throws IOException {
        String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" + File.separator + executable;
        File exeFile = new File(fullExeName);
        if (!exeFile.exists()) {
            throw new IOException("Could not locate executable " + fullExeName + " in the Hadoop binaries.");
        } else {
            return exeFile.getCanonicalPath();
        }
    }

問題解決
分析發現是hadoop.home.dir 為null 和獲取HADOOP_HOME 變數沒找到目錄沒找到,
解決辦法2種:
第一種方法:新增環境變數HADOOP_HOME
第二種方法:直接java設定hadoop.home.dir 指向 winutil.exe 的目錄
主要說一下設定hadoop.home.dir 目錄指向:

        System.setProperty("hadoop.home.dir", "C:\\Program Files\\Hadoop\\hadoop2.7.2-bin");

        //構造一個訪問指定hdfs系統的客戶端物件
        FileSystem fs = FileSystem.get(URI.create("hdfs://192.168.119.129:9000"), new Configuration(), "root");

winutils.exe下載地址:
winutils.exe下載地址