Yes, you can do it using output commiters. Output Committers Hadoop makes sure a job either succeds or fails gracefully. This is done via OutputCommitter. This is accessible from OutputFormat by OutputFormat.getOutputCommiter() public abstract OutputCommitter [...]

Remove from  mapred.exclude Remove from  hdfs.exclude $ hadoop mradmin -refreshNodes $ hadoop dfsadmin -refreshNodes $ hadoop-daemon.sh start tasktracker $ hadoop-daemon.sh start datanode

If your cluster does not have excludes file, add it in hdfs-site.xml dfs.hosts.exclude /usr/local/hadoop/conf/excludes Names a file that contains a list of hosts excluded from cluster   Add hostname of the node you want to remove to  mapred.exclude [...]

Intermediate data is not written in hdfs but in local disk.

If all replicas of one or more blocks of a file become unavailable, a file is considered corrupt and any attempt to access this file will lead to exception. To check health of hadoop filesystem like Linux hadoop has "fsck" command.  fsck generates a summary report that lists the [...]

Hadoop stores data in form of blocks. A block is replicated across the cluster as per the replicationFactor which is 3 by default. Default block size is 64MB.  A file is divided into blocks when it's moved into the cluster. A file can be divided into multiple blocks but one block can [...]

one easy way to differentiate between Hadoop old api and new api is packages. old api packages are identifiable by  mapred or to put it precisely subpackages of org.apache.hadoop.mapred package, new api packages are identifiable by  mapreduce or to put precisely subpackages of [...]

With so many Hadoop versions floating around , it becomes a challenge which one to choose. Choice is simple though, always go for most stable version which at the time of this writing is 1.0.4.  Yarn which is also called Hadoop 2 is making progress towards first stable release. Yarn [...]

Hadoop write path is slightly more complicated than read path.  First client library sends a request to namenode with a named file. After checking permissions NameNode creates filesystem metadata for the file. No blocks are created yet.  Response to client tells that request to [...]

The default size of Distributed Cache for Hadoop is 10 GB but it can be configured using "local.cache.size" in core-site.xml or mapred-site.xml. It's default location is   /tmp/hadoop-root/mapred/local/ archive. How often this cleanup of cache ( if needed) is triggered. By default [...]