Creating Your First Map Reduce Programme

Opening the New Java Project wizard

The New Java Project wizard can be used to create a new java project. There are many ways to open this wizard:

• By clicking on the File menu and choosing New > Java Project
• By right clicking anywhere in the Project Explorer and selecting New > Java Project
• By clicking on the New button ( ) in the Tool bar and selecting Java Project

Using the New Java Project wizard

The New Java Project Wizard has two pages.
On the first page:

• Enter the Project Name
• Select the Java Runtime Environment (JRE) or leave it at the default
• Select the Project Layout which determines whether there would be a separate folder for the sources code and class files. The recommended option is to create separate folders for sources and class files.
  
  You can click on the Finish button to create the project or click on the Next button to change the java build settings.
  On the second page you can change the Java Build Settings like setting the Project dependency (if there are multiple projects) and adding additional jar files to the build path.
  
  

   Writing the Mapper Class

  As we all start up with writing some basic code for map reduce hence we will write a Word Count program which will simply count the number of words in a file and give a out put.
  
  Now here in the mapper class we write WordCountMapper
  
  
  

  package com.hadoop.training;
  
  import java.io.IOException;
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Mapper;
  import java.util.StringTokenizer;
  
  public class WordCountMapper  extends Mapper<LongWritable,Text,Text,IntWritable> {
  
  private final static IntWritable one = new IntWritable(1);
  private Text word = new Text();
  
  public void map (LongWritable key,Text value, Context context) throws IOException,InterruptedException {
  
  StringTokenizer itr = new  StringTokenizer(value.toString());
  
  while (itr.hasMoreTokens()){
  word.set(itr.nextToken());
  context.write(word,one);
  
  }
  
  }
  
  }

  
  

  Writing the Reducer Class 

  Now here in the reducer class we write WordCountReducer
   
  

  package com.hadoop.training;
  
  import java.io.IOException;
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Reducer;
  
  public class  WordCountReducer extends Reducer<Text,IntWritable,Text,IntWritable>{
  
      private IntWritable result = new IntWritable();
  
  public void reduce(Text key,Iterable<IntWritable> value, Context context) throws IOException,InterruptedException {
   int sum = 0;
  for (IntWritable val : value) {
  sum +=val.get();
  
  }
  result.set(sum);
   context.write(key,result);
  }
  
  }

  
  
  

  Writing the MapReduce driver class

  
  Writing the MapReduce driver class as WordCount
  
  

  package com.hadoop.training;
  
  import org.apache.hadoop.fs.Path;
  import org.apache.hadoop.io.IntWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Job;
  import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  
  public class WordCount {
  
  public static void main (String[] args) throws Exception {
  
   if (args.length != 2) {
  System.err.println("Usage: MaxTemperature <input path> <output path>");
  System.exit(-1);
  }
  
  @SuppressWarnings("deprecation")
  Job job = new Job();
  job.setJarByClass(WordCount.class);
  job.setJobName("Word Count");
  
   FileInputFormat.addInputPath(job, new Path(args[0]));
   FileOutputFormat.setOutputPath(job, new Path(args[1]));
  
  job.setMapperClass(WordCountMapper.class);
  job.setReducerClass(WordCountReducer.class);
  
  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(IntWritable.class);
  
   System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
  }

  
  

  Running The Map Reduce programme

  
  $ hadoop jar WC.jar com.hadoop.training.WordCount hdfs://localhost:8020/user/rajeev/input hdfs://localhost:8020/user/rajeev/output