1. 程式人生 > >如何使程式在eclipse上Run As,在叢集中Run

如何使程式在eclipse上Run As,在叢集中Run

一、First Step

將配置檔案拿出來包括(core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml)

注意:修改配置檔案!把主機名改成對應的埠號(或者在window下設定hosts配置主機名和ip的對映

拿出來之後放到你所要執行類的src下!如圖所示:

因為我要執行WordCount類,所以我將這四個配置檔案放入到了此專案的src下。

二、Second Step

在你所寫的(WordCount.java)類中的主方法裡新增一下內容:

如圖:

其中:
Path pathInput = new Path("/betty");
Path pathOutput = new Path("/betty/output");
可以改為:
Path pathInput = new Path(args[0]);
Path pathOutput = new Path(args[1]);
但是要將路徑寫到 Run As ----Run Configurations---Arguments中,將不同路徑空格分開。如圖:
注意:如果沒有Run Configurations就先將程式Run As以下,就會出現。

還要檢查一下Main中包和專案的資訊是否為你所要執行的專案資訊,如果不是改過來。如圖:
注意先檢查左邊框框中的Java Application中是否有你所要執行的類的名字,沒有的話,可以嘗試點選,該框框上方的第一個圖示。 

 

 

其實是添加了以下兩條內容:

1、conf.set("mapreduce.app-submission.cross-platform", "true");

注意:!!此條內容要放在(Job job = Job.getInstance(conf, "word count");)前邊(如圖所示),否則會報以下錯誤

Exception message: /bin/bash: line 0: fg: no job control

Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

因為Job.getInstance(conf,"word count");是從Configuration中獲取的資訊,所以在獲取資訊之前,一定要將此conf資訊更改好,否則你獲取的conf資訊還是和沒有conf.set("mapreduce.app-submission.cross-platform", "true")時的資訊一樣。
或者是你可以直接去mapred-site.xml中更改mapreduce.app-submission.cross-platform此屬性為true,如下:

<property>
  <description>If enabled, user can submit an application cross-platform
  i.e. submit an application from a Windows client to a Linux/Unix server or
  vice versa.
  </description>
  <name>mapreduce.app-submission.cross-platform</name>
  <value>true</value>
</property>

2、job.setJar("mc.jar");

注意:!!!此項括號內的字串名稱和Third Step有關

三、Third Step

將mrmission此專案打成jar包,放在專案下(請注意是專案下,也就是此圖中的mrmission下)!

打包步驟:選中你所要打包的專案--右鍵---Export---Java--JAR File---next---JAR file(設定生成jar file的位置)--next---next---main class(選擇你所要執行的類--我的選擇為WordCount)--finish

四、點選Run As --->選擇--->Run On Hadoop--->執行成功。

五、完整程式碼

package com.hyxy.mrmission;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {
	public static class MyMapper extends Mapper<Object, Text, Text, IntWritable> {
		Text word = new Text();
		final IntWritable one = new IntWritable(1);
		@Override
		protected void map(Object key, Text value, Context context) throws IOException, InterruptedException {
			StringTokenizer itr = new StringTokenizer(value.toString());
			while (itr.hasMoreTokens()) {
				word.set(itr.nextToken());
				context.write(word, one);
			}
		}
	}

	public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
		final IntWritable result = new IntWritable();
		@Override
		protected void reduce(Text key, Iterable<IntWritable> values, Context context)
				throws IOException, InterruptedException {
			int sum = 0;
			for (IntWritable val : values) {
				sum += val.get();
			}
			result.set(sum);
			context.write(key, result);
		}
	}

	public static void main(String[] args) throws Exception {
		Configuration conf = new Configuration();
		conf.set("mapreduce.app-submission.cross-platform", "true");
		Job job = Job.getInstance(conf, "word count");
		job.setJarByClass(WordCount.class);
		job.setJar("mc.jar");
		job.setMapperClass(MyMapper.class);
		job.setReducerClass(MyReducer.class);
		job.setOutputKeyClass(Text.class);
		job.setOutputValueClass(IntWritable.class);
		FileInputFormat.addInputPath(job, new Path("/betty"));
		FileOutputFormat.setOutputPath(job, new Path("/betty/output"));
		System.exit(job.waitForCompletion(true) ? 0 : 1);
	}
}