Login with  Log in with facebook
Hiring Manager? SIGN UP HERE
Apr/13

29

Hardware configuration for a poc/test hadoop cluster

 

Daemons

Before going deeper into hardware configuration, let's first see what different daemons are there in hadoop. 

  • NameNode
  • SecondaryNameNode
  • JobTracker
  • DataNode
  • TaskTracker
In a small cluster, NN, SNN and JT are on same machine. DN and TT are always on same machine, small cluster or big.
 
Let's say we are setting up a small cluster of 5 nodes. One node will host NN,SNN and JT and rest of the 4 will host DN and TT. 
 
node1 needs a lot of memory but every little disk space. Min hardware configuration:
16 GB RAM
1 TB Harddisk
 
node 2-5 need to be higher on disk space. Since we are focusing on easily available machines, let's use following configuration
8 GB RAM
2 TB Harddisk
 
Since depends heavily on network for data transfer, it's recommended to use gigabit ethernet. I would not recommend 10 gig ethernet as that would skyrocket the cost of your test cluster.

 

Schedule a Demo

Schedule a Demo with us

Name *
Email *
Phone *
Company *
Details