Custom Key in map reduce

You can create your custom key in your map reduce program.

Just implement WritableComparable interface.

Steps -

1. Create one class.
2. Extend it from WritableComparable interface.
1. Refernce  - https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/io/WritableComparable.html
3. Override and implement compareTo method.
4. Override and implement equals method.
5. Override and implement hashCode method.
6. Override and implement readFields method.
7. Override and implement write method.

Study More - http://hadooptutorial.info/creating-custom-hadoop-writable-data-type/

Big Data - Java program

Steps to create a java program for map reduce

1. Open eclipse
2. Create java project
3. Add package
4. Add .java files(drive, mapper, reducer, combiner, partitioner, etc)
5. Open project properties
6. Add External jars/libs
1. HadoopHome/common/
2. HadoopHome/common/lib
3. HadoopHome/hdfs
4. HadoopHome/mapreduce
7. Complile
8. Export jar

How to run ?

Create input directory

$hdfs dfs -mkdir /input

Add files in input directory from local file system

$hdfs dfs -put  input.txt /input

or 

Create there

$hdfs dfs -touch  /input/input.txt 

To run use following command

$yarn your_mapreduce_project.jar <packagename>.<classname> /input /output

Check output

$hdfs dfs -ls /output

$hdfs dfs -cat /output/part-r-0000

Big Data Material

Exercises -

http://dcs.gla.ac.uk/~richardm/SICSA-Labsheet-final.pdf



Partitioner Example -
http://www.tutorialspoint.com/map_reduce/map_reduce_partitioner.htm


Iterative MapReduce and Counters
https://hadooptutorial.wikispaces.com/Iterative+MapReduce+and+Counters#driver



https://developer.yahoo.com/hadoop/tutorial/


http://www.codeproject.com/Articles/869383/Implementing-Join-in-Hadoop-Map-Reduce

http://analyticspro.org/author/shantanudeo/

Books - 

1. Hadoop The definitive guide by Tom White
2. Hadoop in practice by Alex Holmes