1. 程式人生 > >Hadoop-mapreduce 程式在windows上執行需要注意的問題

Hadoop-mapreduce 程式在windows上執行需要注意的問題

1.在主程式中需要新增這幾個引數配置

Configuration conf = new Configuration();
        // 1、設定job執行時要訪問的預設檔案系統
        conf.set("fs.defaultFS", HADOOP_ROOT_PATH);
        // 2、設定job提交到哪去執行
        conf.set("yarn.resourcemanager.hostname", "hadoop1");
        conf.set("mapreduce.framework.name", "yarn"); 
        // 3、如果要從windows系統上執行這個job提交客戶端程式,則需要加這個跨平臺提交的引數
conf.set("mapreduce.app-submission.cross-platform", "true"); Job job = Job.getInstance(conf); // 1、封裝引數:jar包所在的位置 job.setJar("D:\\HadoopStudy\\Workspace\\hdfs24\\hdfs24_fat.jar"); //job.setJarByClass(WordCountMain.class); // 2、封裝引數: 本次job所要呼叫的Mapper實現類、Reducer實現類
job.setMapperClass(WordCountMapper.class); job.setReducerClass(WordcountReducer.class); // 3、封裝引數:本次job的Mapper實現類、Reducer實現類產生的結果資料的key、value型別 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.
class); job.setOutputValueClass(IntWritable.class); // 4、封裝引數:本次job要處理的輸入資料集所在路徑、最終結果的輸出路徑 Path output = new Path(HADOOP_OUTPUT_PATH); FileSystem fs = FileSystem.get(new URI(HADOOP_ROOT_PATH), conf); if (fs.exists(output)) { fs.delete(output, true); } FileInputFormat.setInputPaths(job, new Path(HADOOP_INPUT_PATH)); FileOutputFormat.setOutputPath(job, output); // 注意:輸出路徑必須不存在 // 5、封裝引數:想要啟動的reduce task的數量 job.setNumReduceTasks(2); // 6、提交job給yarn boolean res = job.waitForCompletion(true); System.out.println("OK"); System.exit(res ? 0 : -1);

2.需要配置window中的hosts 檔案(識別yarn程式配置的主機ip:我在yarn程式中配置的是hadoop1)

修改打  C盤 ->  Windows -> System32 -> drives -> etc -> hosts檔案

在最下面  增加一行 

10.1.7.96 hadoop1

如果沒有會報以下錯誤:

Exception in thread "main" java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "hadoop1":8032; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
    at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:445)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
    at org.apache.hadoop.ipc.Client.call(Client.java:1373)
    at org.apache.hadoop.ipc.Client.call(Client.java:1337)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy13.getNewApplication(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:217)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
    at com.sun.proxy.$Proxy14.getNewApplication(Unknown Source)
    at org.apache.hadoop.yarn.cl