Hadoop指南：分析一个MapReduce实例

先来看一下这个程序：

package  com.apress.hadoopbook.examples.ch2;

import java.io.IOException;

import  org.apache.hadoop.io.Text;

import  org.apache.hadoop.mapred. FileInputFormat ;

import  org.apache.hadoop.mapred.FileOutputFormat;

import  org.apache.hadoop.mapred.JobClient;

import  org.apache.hadoop.mapred. JobConf ;

import  org.apache.hadoop.mapred.KeyValueTextInputFormat;

import  org.apache.hadoop.mapred.RunningJob;

import  org.apache.hadoop.mapred.lib. IdentityMapper ;

import  org.apache.hadoop.mapred.lib. IdentityReducer ;

import  org.apache.log4j.Logger;

public  class  MapReduceIntro {

protected  static Loggerlogger= Logger.getLogger(MapReduceIntro.class);

public static  voidmain(final  String[] args) {

try  {

final  JobConf conf = new  JobConf(MapReduceIntro.class);

//定义Hadoop运行作业时的临时文件目录

conf.set("hadoop.tmp.dir","/tmp");

MapReduceIntroConfig.exampleHouseKeeping(conf, MapReduceIntroConfig.getInputDirectory(), MapReduceIntroConfig.getOutputDirectory());

conf.setInputFormat(KeyValueTextInputFormat.class);

FileInputFormat .setInputPaths(conf, MapReduceIntroConfig.getInputDirectory());

conf.setMapperClass( IdentityMapper .class);

FileOutputFormat.setOutputPath(conf, MapReduceIntroConfig.getOutputDirectory());

conf.setOutputKeyClass(Text.class);

conf.setOutputValueClass(Text.class);

conf.setNumReduceTasks(1);

conf.setReducerClass( IdentityReducer .class);

logger.info("Launching the job.");

final  RunningJob job = JobClient.runJob(conf);

logger.info("The job has completed.");

if  (!job.isSuccessful()) {

logger.error("The job failed.");

System.exit(1);

}

logger.info("The job completed successfully.");

System.exit(0);

} catch  (final  IOException e) {

logger.error("The job has failed due to an IO Exception", e);

e.printStackTrace();

}

}

}

程序很短，而且注释很详细，不过我们还是来看看它有哪些部分组成。

1.日志跟踪。通过日志我们才能跟踪 MapReduceIntro 类，将作业运行期间所发生的动作记录下来，便于后面的分析：

protected static Loggerlogger= Logger.getLogger(MapReduceIntro.class);