$ java -versionIf everything works fine it will give you the following output.
java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b13) Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)If java is not installed in your system, then follow the given steps for installing java.
$ cd Downloads/ $ ls jdk-7u71-Linux-x64.gz $ tar zxf jdk-7u71-Linux-x64.gz $ ls jdk1.7.0_71 jdk-7u71-Linux-x64.gz
$ su password: # mv jdk1.7.0_71 /usr/local/ # exit
export JAVA_HOME=/usr/local/jdk1.7.0_71 export PATH=PATH:$JAVA_HOME/binNow verify the java -version command from the terminal as explained above. Follow the above process and install java in all your cluster nodes.
# useradd hadoop # passwd hadoop
# vi /etc/hosts enter the following lines in the /etc/hosts file. 192.168.1.109 hadoop-master 192.168.1.145 hadoop-slave-1 192.168.56.1 hadoop-slave-2
# su hadoop $ ssh-keygen -t rsa $ ssh-copy-id -i ~/.ssh/id_rsa.pub tutorialspoint@hadoop-master $ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp1@hadoop-slave-1 $ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop_tp2@hadoop-slave-2 $ chmod 0600 ~/.ssh/authorized_keys $ exit
# mkdir /opt/hadoop # cd /opt/hadoop/ # wget http://apache.mesi.com.ar/hadoop/common/hadoop-1.2.1/hadoop-1.2.0.tar.gz # tar -xzf hadoop-1.2.0.tar.gz # mv hadoop-1.2.0 hadoop # chown -R hadoop /opt/hadoop # cd /opt/hadoop/hadoop/
<configuration> <property> <name>fs.default.name</name> <value>hdfs://hadoop-master:9000/</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> </configuration>
<configuration> <property> <name>dfs.data.dir</name> <value>/opt/hadoop/hadoop/dfs/name/data</value> <final>true</final> </property> <property> <name>dfs.name.dir</name> <value>/opt/hadoop/hadoop/dfs/name</value> <final>true</final> </property> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
<configuration> <property> <name>mapred.job.tracker</name> <value>hadoop-master:9001</value> </property> </configuration>
export JAVA_HOME=/opt/jdk1.7.0_17 export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf
# su hadoop $ cd /opt/hadoop $ scp -r hadoop hadoop-slave-1:/opt/hadoop $ scp -r hadoop hadoop-slave-2:/opt/hadoop
# su hadoop $ cd /opt/hadoop/hadoop
$ vi etc/hadoop/masters hadoop-master
$ vi etc/hadoop/slaves hadoop-slave-1 hadoop-slave-2
# su hadoop $ cd /opt/hadoop/hadoop $ bin/hadoop namenode –format
11/10/14 10:58:07 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = hadoop-master/192.168.1.109 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May 6 06:59:37 UTC 2013 STARTUP_MSG: java = 1.7.0_71 ************************************************************/ 11/10/14 10:58:08 INFO util.GSet: Computing capacity for map BlocksMap editlog=/opt/hadoop/hadoop/dfs/name/current/edits …………………………………………………. …………………………………………………. …………………………………………………. 11/10/14 10:58:08 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted. 11/10/14 10:58:08 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop-master/192.168.1.15 ************************************************************/
$ cd $HADOOP_HOME/sbin $ start-all.sh
IP address : 192.168.1.103 netmask : 255.255.255.0 hostname : slave3.in
useradd hadoop
passwd hadoop
Setup Password less connectivity from master to new slave.mkdir -p $HOME/.ssh chmod 700 $HOME/.ssh ssh-keygen -t rsa -P '' -f $HOME/.ssh/id_rsa cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys chmod 644 $HOME/.ssh/authorized_keys Copy the public key to new slave node in hadoop user $HOME directory scp $HOME/.ssh/id_rsa.pub hadoop@192.168.1.103:/home/hadoop/
su hadoop ssh -X hadoop@192.168.1.103Copy the content of public key into file "$HOME/.ssh/authorized_keys" and then change the permission for the same by executing the following commands.
cd $HOME mkdir -p $HOME/.ssh chmod 700 $HOME/.ssh cat id_rsa.pub >>$HOME/.ssh/authorized_keys chmod 644 $HOME/.ssh/authorized_keysCheck ssh login from the master machine. Now check if you can ssh to the new node without a password from the master.
ssh hadoop@192.168.1.103 or hadoop@slave3
On new slave3 machine NETWORKING=yes HOSTNAME=slave3.inTo make the changes effective, either restart the machine or run hostname command to a new machine with the respective hostname (restart is a good option).
192.168.1.102 slave3.in slave3Now try to ping the machine with hostnames to check whether it is resolving to IP or not.
ping master.in
su hadoop or ssh -X hadoop@192.168.1.103
./bin/hadoop-daemon.sh start datanode
$ jps 7141 DataNode 10312 Jps
$ su hadoop
<property> <name>dfs.hosts.exclude</name> <value>/home/hadoop/hadoop-1.2.1/hdfs_exclude.txt</value> <description>DFS exclude</description> </property>
slave2.in
$ $HADOOP_HOME/bin/hadoop dfsadmin -refreshNodesThis will force the NameNode to re-read its configuration, including the newly updated ‘excludes’ file. It will decommission the nodes over a period of time, allowing time for each node's blocks to be replicated onto machines which are scheduled to remain active.
$ $HADOOP_HOME/bin/hadoop dfsadmin -report
$ $HADOOP_HOME/bin/hadoop-daemon.sh stop tasktracker $HADOOP_HOME/bin/hadoop-daemon.sh
!/usr/bin/python import sys # Input takes from standard input for myline in sys.stdin: # Remove whitespace either side myline = myline.strip() # Break the line into words words = myline.split() # Iterate the words list for myword in words: # Write the results to standard output print '%s\t%s' % (myword, 1)Make sure this file has execution permission (chmod +x /home/ expert/hadoop-1.2.1/mapper.py).
#!/usr/bin/python from operator import itemgetter import sys current_word = "" current_count = 0 word = "" # Input takes from standard input for myline in sys.stdin: # Remove whitespace either side myline = myline.strip() # Split the input we got from mapper.py word, count = myline.split('\t', 1) # Convert count variable to integer try: count = int(count) except ValueError: # Count was not a number, so silently ignore this line continue if current_word == word: current_count += count else: if current_word: # Write result to standard output print '%s\t%s' % (current_word, current_count) current_count = count current_word = word # Do not forget to output the last word if needed! if current_word == word: print '%s\t%s' % (current_word, current_count)Save the mapper and reducer codes in mapper.py and reducer.py in Hadoop home directory. Make sure these files have execution permission (chmod +x mapper.py and chmod +x reducer.py). As python is indentation sensitive so the same code can be download from the below link.
$ $HADOOP_HOME/bin/hadoop jar contrib/streaming/hadoop-streaming-1. 2.1.jar \ -input input_dirs \ -output output_dir \ -mapper <path/mapper.py \ -reducer <path/reducer.pyWhere "\" is used for line continuation for clear readability.
./bin/hadoop jar contrib/streaming/hadoop-streaming-1.2.1.jar -input myinput -output myoutput -mapper /home/expert/hadoop-1.2.1/mapper.py -reducer /home/expert/hadoop-1.2.1/reducer.py
| Parameters | Description |
|---|---|
| -input directory/file-name | Input location for mapper. (Required) |
| -output directory-name | Output location for reducer. (Required) |
| -mapper executable or script or JavaClassName | Mapper executable. (Required) |
| -reducer executable or script or JavaClassName | Reducer executable. (Required) |
| -file file-name | Makes the mapper, reducer, or combiner executable available locally on the compute nodes. |
| -inputformat JavaClassName | Class you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default. |
| -outputformat JavaClassName | Class you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default. |
| -partitioner JavaClassName | Class that determines which reduce a key is sent to. |
| -combiner streamingCommand or JavaClassName | Combiner executable for map output. |
| -cmdenv name=value | Passes the environment variable to streaming commands. |
| -inputreader | For backwards-compatibility: specifies a record reader class (instead of an input format class). |
| -verbose | Verbose output. |
| -lazyOutput | Creates output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to output.collect (or Context.write). |
| -numReduceTasks | Specifies the number of reducers. |
| -mapdebug | Script to call when map task fails. |
| -reducedebug | Script to call when reduce task fails. |
| Input | Output | |
|---|---|---|
| Map | <k1, v1> | list (<k2, v2>) |
| Reduce | <k2, list(v2)> | list (<k3, v3>) |
| Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec | Avg | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1979 | 23 | 23 | 2 | 43 | 24 | 25 | 26 | 26 | 26 | 26 | 25 | 26 | 25 |
| 1980 | 26 | 27 | 28 | 28 | 28 | 30 | 31 | 31 | 31 | 30 | 30 | 30 | 29 |
| 1981 | 31 | 32 | 32 | 32 | 33 | 34 | 35 | 36 | 36 | 34 | 34 | 34 | 34 |
| 1984 | 39 | 38 | 39 | 39 | 39 | 41 | 42 | 43 | 40 | 39 | 38 | 38 | 40 |
| 1985 | 38 | 39 | 39 | 39 | 39 | 41 | 41 | 41 | 00 | 40 | 39 | 39 | 45 |
1979 23 23 2 43 24 25 26 26 26 26 25 26 25 1980 26 27 28 28 28 30 31 31 31 30 30 30 29 1981 31 32 32 32 33 34 35 36 36 34 34 34 34 1984 39 38 39 39 39 41 42 43 40 39 38 38 40 1985 38 39 39 39 39 41 41 41 00 40 39 39 45
package hadoop; import java.util.*; import java.io.IOException; import java.io.IOException; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; public class ProcessUnits { //Mapper class public static class E_EMapper extends MapReduceBase implements Mapper<LongWritable ,/*Input key Type */ Text, /*Input value Type*/ Text, /*Output key Type*/ IntWritable> /*Output value Type*/ { //Map function public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = value.toString(); String lasttoken = null; StringTokenizer s = new StringTokenizer(line,"\t"); String year = s.nextToken(); while(s.hasMoreTokens()) { lasttoken=s.nextToken(); } int avgprice = Integer.parseInt(lasttoken); output.collect(new Text(year), new IntWritable(avgprice)); } } //Reducer class public static class E_EReduce extends MapReduceBase implements Reducer< Text, IntWritable, Text, IntWritable > { //Reduce function public void reduce( Text key, Iterator <IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int maxavg=30; int val=Integer.MIN_VALUE; while (values.hasNext()) { if((val=values.next().get())>maxavg) { output.collect(key, new IntWritable(val)); } } } } //Main function public static void main(String args[])throws Exception { JobConf conf = new JobConf(Eleunits.class); conf.setJobName("max_eletricityunits"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setMapperClass(E_EMapper.class); conf.setCombinerClass(E_EReduce.class); conf.setReducerClass(E_EReduce.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } }Save the above program as ProcessUnits.java. The compilation and execution of the program is explained below.
$ mkdir units
$ javac -classpath hadoop-core-1.2.1.jar -d units ProcessUnits.java $ jar -cvf units.jar -C units/ .
$HADOOP_HOME/bin/hadoop fs -mkdir input_dir
$HADOOP_HOME/bin/hadoop fs -put /home/hadoop/sample.txt input_dir
$HADOOP_HOME/bin/hadoop fs -ls input_dir/
$HADOOP_HOME/bin/hadoop jar units.jar hadoop.ProcessUnits input_dir output_dirWait for a while until the file is executed. After execution, as shown below, the output will contain the number of input splits, the number of Map tasks, the number of reducer tasks, etc.
INFO mapreduce.Job: Job job_1414748220717_0002 completed successfully 14/10/31 06:02:52 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=61 FILE: Number of bytes written=279400 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=546 HDFS: Number of bytes written=40 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=146137 Total time spent by all reduces in occupied slots (ms)=441 Total time spent by all map tasks (ms)=14613 Total time spent by all reduce tasks (ms)=44120 Total vcore-seconds taken by all map tasks=146137 Total vcore-seconds taken by all reduce tasks=44120 Total megabyte-seconds taken by all map tasks=149644288 Total megabyte-seconds taken by all reduce tasks=45178880 Map-Reduce Framework Map input records=5 Map output records=5 Map output bytes=45 Map output materialized bytes=67 Input split bytes=208 Combine input records=5 Combine output records=5 Reduce input groups=5 Reduce shuffle bytes=6 Reduce input records=5 Reduce output records=5 Spilled Records=10 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=948 CPU time spent (ms)=5160 Physical memory (bytes) snapshot=47749120 Virtual memory (bytes) snapshot=2899349504 Total committed heap usage (bytes)=277684224 File Output Format Counters Bytes Written=40
$HADOOP_HOME/bin/hadoop fs -ls output_dir/
$HADOOP_HOME/bin/hadoop fs -cat output_dir/part-00000Below is the output generated by the MapReduce program.
1981 34 1984 40 1985 45
$HADOOP_HOME/bin/hadoop fs -cat output_dir/part-00000/bin/hadoop dfs get output_dir /home/hadoop
| Options | Description |
|---|---|
| namenode -format | Formats the DFS filesystem. |
| secondarynamenode | Runs the DFS secondary namenode. |
| namenode | Runs the DFS namenode. |
| datanode | Runs a DFS datanode. |
| dfsadmin | Runs a DFS admin client. |
| mradmin | Runs a Map-Reduce admin client. |
| fsck | Runs a DFS filesystem checking utility. |
| fs | Runs a generic filesystem user client. |
| balancer | Runs a cluster balancing utility. |
| oiv | Applies the offline fsimage viewer to an fsimage. |
| fetchdt | Fetches a delegation token from the NameNode. |
| jobtracker | Runs the MapReduce job Tracker node. |
| pipes | Runs a Pipes job. |
| tasktracker | Runs a MapReduce task Tracker node. |
| historyserver | Runs job history servers as a standalone daemon. |
| job | Manipulates the MapReduce jobs. |
| queue | Gets information regarding JobQueues. |
| version | Prints the version. |
| jar <jar> | Runs a jar file. |
| distcp <srcurl> <desturl> | Copies file or directories recursively. |
| distcp2 <srcurl> <desturl> | DistCp version 2. |
| archive -archiveName NAME -p | Creates a hadoop archive. |
| <parent path> <src>* <dest> | |
| classpath | Prints the class path needed to get the Hadoop jar and the required libraries. |
| daemonlog | Get/Set the log level for each daemon |
| GENERIC_OPTIONS | Description |
|---|---|
| -submit <job-file> | Submits the job. |
| status <job-id> | Prints the map and reduce completion percentage and all job counters. |
| counter <job-id> <group-name> <countername> | Prints the counter value. |
| -kill <job-id> | Kills the job. |
| -events <job-id> <fromevent-#> <#-of-events> | Prints the events' details received by jobtracker for the given range. |
| -history [all] <jobOutputDir> - history < jobOutputDir> | Prints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option. |
| -list[all] | Displays all jobs. -list displays only jobs which are yet to complete. |
| -kill-task <task-id> | Kills the task. Killed tasks are NOT counted against failed attempts. |
| -fail-task <task-id> | Fails the task. Failed tasks are counted against |
| failed attempts. | |
| set-priority <job-id> <priority> | Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW |
$ $HADOOP_HOME/bin/hadoop job -status <JOB-ID> e.g. $ $HADOOP_HOME/bin/hadoop job -status job_201310191043_0004
$ $HADOOP_HOME/bin/hadoop job -history <DIR-NAME> e.g. $ $HADOOP_HOME/bin/hadoop job -history /user/expert/output
$ $HADOOP_HOME/bin/hadoop job -kill <JOB-ID> e.g. $ $HADOOP_HOME/bin/hadoop job -kill job_201310191043_0004
"<path>" means any file or directory name. "<path>..." means one or more file or directory names. "<file>" means any filename. "<src>" and "<dest>" are path names in a directed operation. "<localSrc>" and "<localDest>" are paths as above, but on the local file system.All other files and path names refer to the objects inside HDFS.
| 1. |
ls <path> Lists the contents of the directory specified by path, showing the names, permissions, owner, size and modification date for each entry. |
| 2. | lsr <path> Behaves like -ls, but recursively displays entries in all subdirectories of path. |
| 3. | du <path> Shows disk usage, in bytes, for all the files which match path; filenames are reported with the full HDFS protocol prefix. |
| 4. |
dus <path> Like -du, but prints a summary of disk usage of all files/directories in the path. |
| 5. |
mv <src><dest> Moves the file or directory indicated by src to dest, within HDFS. |
| 6. |
cp <src> <dest> Copies the file or directory identified by src to dest, within HDFS. |
| 7. |
rm <path> Removes the file or empty directory identified by path. |
| 8. |
rmr <path> Removes the file or directory identified by path. Recursively deletes any child entries (i.e., files or subdirectories of path). |
| 9. | put <localSrc> <dest> Copies the file or directory from the local file system identified by localSrc to dest within the DFS. |
| 10. | copyFromLocal <localSrc> <dest> Identical to -put |
| 11. | moveFromLocal <localSrc> <dest> Copies the file or directory from the local file system identified by localSrc to dest within HDFS, and then deletes the local copy on success. |
| 12. | get [-crc] <src> <localDest> Copies the file or directory in HDFS identified by src to the local file system path identified by localDest. |
| 13. | getmerge <src> <localDest> Retrieves all files that match the path src in HDFS, and copies them to a single, merged file in the local file system identified by localDest. |
| 14. | cat <filen-ame> Displays the contents of filename on stdout. |
| 15. | copyToLocal <src> <localDest> Identical to -get |
| 16. | moveToLocal <src> <localDest> Works like -get, but deletes the HDFS copy on success. |
| 17. | mkdir <path> Creates a directory named path in HDFS. Creates any parent directories in path that are missing (e.g., mkdir -p in Linux). |
| 18. | setrep [-R] [-w] rep <path> Sets the target replication factor for files identified by path to rep. (The actual replication factor will move toward the target over time) |
| 19. | touchz <path> Creates a file at path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0. |
| 20. | test -[ezd] <path> Returns 1 if path exists; has zero length; or is a directory or 0 otherwise. |
| 21. | stat [format] <path> Prints information about path. Format is a string which accepts file size in blocks (%b), filename (%n), block size (%o), replication (%r), and modification date (%y, %Y). |
| 22. | tail [-f] <file2name> Shows the last 1KB of file on stdout. |
| 23. | chmod [-R] mode,mode,... <path>... Changes the file permissions associated with one or more objects identified by path.... Performs changes recursively with R. mode is a 3-digit octal mode, or {augo}+/-{rwxX}. Assumes if no scope is specified and does not apply an umask. |
| 24. | chown [-R] [owner][:[group]] <path>... Sets the owning user and/or group for files or directories identified by path.... Sets owner recursively if -R is specified. |
| 25. | chgrp [-R] group <path>... Sets the owning group for files or directories identified by path.... Sets group recursively if -R is specified. |
| 26. |
help <cmd-name> Returns usage information for one of the commands listed above. You must omit the leading '-' character in cmd. |