Introduction to MapReduce
MapReduce tutorial covers the introduction to MapReduce, its definition, why MapReduce, algorithms, examples, installation, API (Application Programming interface), implementation of MapReduce, MapReduce Partitioner, MapReduce Combiner, and administration.
Why to use MapReduce?
Initially created by Google, MapReduce soon gained immense popularity due to its unmatched qualities, which mandated big data players to deploy it. Some of its unique features are as follows:
| Features |
Description |
| Flexibility |
Can be developed in any language like Java, c++, python, etc. |
| Scalability |
Able to process petabytes of data on a single cluster |
| Recovery |
Takes care of failure by storing the replica on another machine |
| Lesser data motion |
Processing tasks appear on physical nodes which increases the speed in turn. |
Apart from the above key features some of the key highlights of this technology are:
- Map task stores data into a local disk while Reduce task in HDFS.
- Map tasks are created for each split of equal size which is equal to an HDFS block~ 64 MB
- Tasktracker sends heartbeat signals to notify about the current state.
Last year MapReduce received the first place at “TeraByte Sort Benchmark”. They used 910 nodes, every node with two cores, i.e., a total of 1820 cores and were able to store the entire data in memory across the nodes. By implementation of MapReduce they were able to arrange entire one terabyte of data in 209 seconds. Users program, i.e., map and reduce functions in ANSI C.
Let’s delve further into this MapReduce tutorial.