Intellipaat
Intellipaat

Java Mapreduce Tutorial

Java Mapreduce Tutorial
 137 Views     0 Comment(s)

What is MapReduce?

MapReduce is a programming model useful for processing huge data sets and is also used to divide computing on various computers. This model is activated by mainly two functions: Reduce and Map which are mostly used in functional programming.Function output is based on input data. So, based on the input data, the output is always confirmed. Functions easily guarantee flexibility.

Features of MapReduce:

MapReduce offers framework for MapReduce execution and partial failure of processing cluster is to be expected. MapReduce programming is an independent model which allows data local processing and it also administers communication within the process.

MapReduce has several attributes like height, complexion, which clearly define what they really represent. For example: Nationality is an attribute that represents from where the person is. Looking it from a designer’s perspective, attributes are defined keeping in mind the objects.

For instance, Colour is the attribute to be defined for the objects like Pencil, pen, table, chair, etc. Based on the objects, attributes must be clearly defined which themselves clearly represent characteristics.

When we define the class, we also define attributes. Let’s say if an object is a square, then attributes can be its height, width, length etc that represent that the object is exactly the Square. Class name is important to properly define attributes.

For creating an entity, values are required. These may be eithera variable, string or constant. For instance, colour may be red, blue, green; yellow, which is variable, weight may be in constant numbers say 1, 2 or more.

While writing a programme in Java for defining a class, MapReduce is useful as compared to other programmes like C, C++, While Logo etc.

Three different Parts of MapReduce:

a) Driver: The Driver code works on client machine and is used for developing the task configuration. It contains main() method that welcomes arguments from the command line.

For driver class, some common libraries are included:

importorg.apache.hadoop.fs.Path;
importorg.apache.hadoop.io.*;
importorg.apache.hadoop.mapred.*;

In some cases, the command line parameters are then passed to the Driver program which contains input files and path to output directory. These path locations are from HDFS. If the output location is already available then the programme will cause an error.

Another step for Driver Programme is to develop a job which needs to be presented to the group.

b) Mapper: Mapper code reads the input files as value pairs and the mapper class, then expands MapReduce Base and applies the Mapper interface. It expects four generics that define the type of input and output keys.

c) Reducer: The code is used to interpret the outputs generated by various mappers like Key Value pairs and it increases the MapReduce Base and adopts the reducer interface. Further, this reducer interface has four generics that determine the type of input and output key value pairs.

The first two parameters determine central key and value types while the remaining shows the final output key and value types. The keys are writable comparables, and values are writeable.

Hence, this will definitely make clear what exactly Map Reduce is and why it’s necessary to Learn MapReduce.

 

Related Articles