A lot of friends are currently using or planning to use big data technologies, particularlyHadoop. Hadoop and Hbase are your companion
if you need to crunch large amounts of data in short time and without throwing out a lot of resources (a single box with lots of processing power).
Ideally, you will need a cluster of commodity hardware (cheap) servers, but for development and testing purposes a local instance of Hadoop will do better than a remote server cluster.
This recipe will take you about 15 minutes to have a Hadoop and Hbase instance up and running.
I’m assuming that you know what Hadoop and Hbase are, and that you need to install a development instance of both.
Pre-requisites and Assumptions
I’m assuming that you are installing Hadoop and Hbase on a box running OSX (I’m running OSX 10.8 mountain lion, but I don’t see a reason why this should not work the same way for other versions of OSX).
I’m also assuming that you are using your regular user (in my case “freddy”), that your home directory is /Users/USER (refered
as ~ or $HOME)
and your Hadoop and Hbase installation directory are respectively ~/tools/hd/hadoop and ~/tools/hd/hbase.
If you don’t have a ~/tools/hd directory
you can create it right new by executing the command. mkdir
For the sake of simplicity, in the reminder of this recipe I will use the shell variables $HD_HOME,$HADOOP_HOME and $HBASE_HOME to
identify the Hadoop/Hbase installation directory.
There are two basic requirements that you need to complete before installing Hadoop and Hbase, these are Java and ssh.
Hadoop and Hbase are developed using Java, and therefore to run them you should have Java installed on your system. Verify that you have Java 1.6 or higher installed on your system (by default in OSX 10.8) by executing:
This should produce an output similar to:
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)
If you don’t have Java installed in your system, entering the previous command will prompt its installation (OSX 10.8+ only).
Hadoop manages all its nodes using ssh (yes,
event a single node cluster). To enable ssh (Remote Login) on OSX 10.8 you should execute the following command:
sudo systemsetup -setremotelogin on
You will also need to generate a ssh public/private keypair in order to allow Hadoop to connect password-less into your node(s). To do so, execute the following command:
ssh-keygen -t rsa -P ""
This will produce an output similar to:
Generating public/private rsa key pair.
Enter file in which to save the key (Users/freddy/.ssh/id_rsa): id_rsa
Your identification has been saved in id_rsa_hbase.
Your public key has been saved in id_rsa_hbase.pub.
The key fingerprint is:
46:a9:26:fc:16:c8:ca:f4:64:37:cd:8c:c6:8d:41:3a [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o . |
| E . o |
| o + @ |
| . B @ S |
| o = * + |
| o . o |
| . |
Now install the key into the authorized keys:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
And try to log into localhost (ssh should not ask for a connection password).
If you already have a default ssh keys in your computer (otherwise ignore this alert), you may want to change the name of the generated key. For example enterid_rsa_hbase instead
of id_rsa .
To tell ssh to use this particular key when connecting to localhost, edit the file~/.ssh/config and
add the following lines:
host localhost user freddy identityfile ~/.ssh/id_rsa_hbase
Now, your system is ready to run Hadoop so you can proceed to download it. To do so, you can either go to the Hadoop distribution
site, choose a mirror close to your location and download it (then copu to $HD_HOME),
or execute the following commands:
curl http://mir2.ovh.net/ftp.apache.org/dist/hadoop/common/stable/hadoop-1.1.2.tar.gz > hadoop-1.1.2.tar.gz
mv hadoop-1.1.2.tar.gz $HD_HOME/
tar xvzf hadoop-1.1.2.tar.gz
ln -s hadoop-1.1.2 hadoop
This will download and install the Hadoop files into $HADOOP_HOME.
In order to set up your single node cluster, you will need to modify a series of configuration files (all located under $HADOOP_HOME/conf/).
The first configuration files to modify is hadoop-env.sh.
This file contains all the “shell” environment variables that Hadoop requires to work properly. So fire your prefer text editor and open $HADOOP_HOME/conf/hadoop-env.sh.
Add the lines below to your file.
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/HomeHADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
The property -Djava.net.preferIPv4Stack=true will
force Hadoop components to use IPv4. This is requires since Hadoop does not work pretty well with IPv6. You can find more about this here and here.
The property -Djava.security.krb5.realm=
-Djava.security.krb5.kdc= indicates Hadoop to avoid kerberos realm. Without this line you will get the following warningUnable
to load realm info from SCDynamicStore. You can find more about thishere.
The next file you want to modify is core-site.xml.
This file contains the base configuration of Hadoop distributed file system (HDFS). The property fs.default.name indicates
the host and port where the Hadoop name node is be located. The property hadoop.tmp.dir indicates
Hadoop where it should write its distributed systems temporal (and non-temporal) files (in this case $HD_HOME/hadoopdata).
<configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/Users/freddy/tools/hd/hadoopdata</value> <description>A base for other temporary directories.</description> </property></configuration>
Hadoop must be able to write to the hadoop.tmp.dir directory
HDFS uses a replication procedure to ensure fault tolerance (and avoid losing data). The configuration file hdfs-site.xml contains
the parameters that rule this “replication” (a list of other properties that can be included in this file can be found here).
Notice that since we have only a single node, it’s fair to have only one copy of each file.
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>
Hadoop enables parallel processing using MapReduce. The Hadoop MapReduce
instance is manager by the property file mapred-site.xml.
Here you can set several properties, but to make MapReduce work only a few are requires. Set mapred.job.tracker to
indicate the host and port where the job tracker is located (the “task manager”).mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximumindicate
the maximum number of map and reduce tasks that can be executed.
As a rule of thumb, you should not have map tasks than the number of cores in your cpu (single node cluster), and one reduce for each physical disk you have. In my case I have four cores and one disk.
<configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>4</value> </property> <property> <name>mapred.tasktracker.reduce.tasks.maximum</name> <value>1</value> </property></configuration>
Format your name node
Before you launch Hadoop, HDFS requires to be initialized (formatted). To format your HDFS name node execute the following command:
$HADOOP_HOME/bin/hadoop namenode -format
You will get a very verbose output from this command, verify that you have a line similar to:
13/06/09 19:18:51 INFO common.Storage: Storage directory /Users/freddy/tools/hd/hadoopdata/dfs/name has been successfully formatted.
Your Hadoop instance is configured and ready to be launched.
A command that comes at hand to check what (java) services are running is jps.
If you execute it on your console you should see an output similar to:
If JobTracker, NameNode, SecondaryNameNode, DataNode, and TaskTracker are there, then Hadoop is running.
Note that HDFS is not a filesystem layer but a “virtual” file system that provides a very particular API. Try to browse the filesystem root:
$HADOOP_HOME/bin/hadoop dfs -ls /
create a new folder:
$HADOOP_HOME/bin/hadoop dfs -mkdir /test
drwxr-xr-x - freddy supergroup 0 2013-06-09 19:26 /test
copy a file:
echo "test file" > /tmp/tempFileTest
$HADOOP_HOME/bin/hadoop dfs -copyFromLocal /tmp/tempFileTest /test
and check the file content.
$HADOOP_HOME/bin/hadoop dfs -ls /test
-rw-r--r-- 1 freddy supergroup 0 2013-06-09 19:30 /test/tempFileTest
$HADOOP_HOME/bin/hadoop dfs -cat /test/tempFileTest
You may also want to check that your brand new MapReduce setup is working.
$HADOOP_HOME/bin/hadoop jar hadoop-examples-1.1.2.jar pi 10 50
This job should output something similar to:
Job Finished in 20.456 seconds
Estimated value of Pi is 3.16000000000000000000
Hadoop comes with a few web-interfaces that you may want to browse: http://localhost:50070HDFS webui
http://localhost:50030 MapReduce webui
http://localhost:50060 Task Tracker webui
Sometimes it’s useful to start and stop each Hadoop service separately. Under$HADOOP_HOME/bin you
will find a series of scripts dedicated to starting and stopping individual services. Their name will hint the services they start or stop.
Now that you have successfully setup and launch Hadoop it’s time to install Hbase. Similarly to Hadoop, you have two options to get Hbase. You can either go to the Hbase distribution
site, choose a mirror close to your location and download it (then copy to $HD_HOME),
or execute the following commands:
curl http://apache.websitebeheerjd.nl/hbase/stable/hbase-0.94.8.tar.gz > hbase-0.94.8.tar.gz
mv hbase-0.94.8.tar.gz $HD_HOME/
tar xvzf hbase-0.94.8.tar.gz
ln -s hbase-0.94.8 hbase
Configuring Hbase is quite easy (a very basic instance), you need to modify only two files (located under $HBASE_HOME/conf).
The file hbase-env.sh sets the execution
environment for Hbase. This file works the same way with as hadoop-env.sh for
Hadoop. Add the following lines to hbase-env.sh:
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/HomeHBASE_OPTS="-Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
Hbase properties are governed by the file hbase-site.xml.
The only configuration parameter that you need to specify to make Hbase work is hbase.rootdir,
the Hbase root directory. This directory can be either a local file file:/// or
an HDFS instancehdfs://. In this
particular case we are pointing Hbase to our newly installed HDFS instance. Other properties that can be set in this files can be found here.
Hbase requires Zookeper to work. By default Hbase comes with an embedded instance of Zookeeper, which relieves us from the task of
setting one by ourselves. In the case that you may want to know more about Zookeper, its configuration, and its role on the Hbase architecture checkout this article.
<configuration> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:9000/hbase</value> </property></configuration>
Now you are ready to launch with Hbase. To start Hbase just execute the following command:
In order to test your Hbase installation, launch the Hbase shell and play with it (heavily inspired from http://hbase.apache.org/book/quickstart.html).
To launch the Hbase shell execute the following command:
You should be prompted to the Hbase interactive interpreter:
HBase Shell; enter 'help' for list of supported commands.
Type "exit" to leave the HBase Shell
Version 0.94.8, r1485407, Wed May 22 20:53:13 UTC 2013
Create a new table and put new values on it:
hbase(main):003:0> create 'test', 'cf'
0 row(s) in 1.2200 seconds
hbase(main):003:0> list 'test'
1 row(s) in 0.0550 seconds
hbase(main):004:0> put 'test', 'row1', 'cf:a', 'value1'
0 row(s) in 0.0560 seconds
hbase(main):005:0> put 'test', 'row2', 'cf:b', 'value2'
0 row(s) in 0.0370 seconds
hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3'
0 row(s) in 0.0450 seconds
scan the table values:
hbase(main):007:0> scan 'test'
row1 column=cf:a, timestamp=1288380727188, value=value1
row2 column=cf:b, timestamp=1288380738440, value=value2
row3 column=cf:c, timestamp=1288380747365, value=value3
3 row(s) in 0.0590 seconds
get a value through its key:
hbase(main):008:0> get 'test', 'row1'
cf:a timestamp=1288380727188, value=value1
1 row(s) in 0.0400 seconds
disable and drop (delete) the table.
hbase(main):012:0> disable 'test'
0 row(s) in 1.0930 seconds
hbase(main):013:0> drop 'test'
0 row(s) in 0.0770 seconds
If you could execute those commands successfully then your hbase instance is working properly.
http://localhost:60010/ Hbase master webui
http://localhost:60030/ Hbase region server webui
Now that your development instance of Hadoop and Hbase are up and running you may want to start creating a few MapReduce/Hbase apps.