The document provides steps to install Hadoop on a single node and run a sample MapReduce job. The key steps are:
1. Downloading and extracting the Hadoop tar file.
2. Configuring passwordless SSH and setting environment variables like JAVA_HOME.
3. Editing configuration files like core-site.xml and hdfs-site.xml.
4. Formatting the namenode and starting the Hadoop daemons.
5. Running a sample word count MapReduce job to count words in a file and viewing the output.
The document provides steps to install Hadoop on a single node and run a sample MapReduce job. The key steps are:
1. Downloading and extracting the Hadoop tar file.
2. Configuring passwordless SSH and setting environment variables like JAVA_HOME.
3. Editing configuration files like core-site.xml and hdfs-site.xml.
4. Formatting the namenode and starting the Hadoop daemons.
5. Running a sample word count MapReduce job to count words in a file and viewing the output.
The document provides steps to install Hadoop on a single node and run a sample MapReduce job. The key steps are:
1. Downloading and extracting the Hadoop tar file.
2. Configuring passwordless SSH and setting environment variables like JAVA_HOME.
3. Editing configuration files like core-site.xml and hdfs-site.xml.
4. Formatting the namenode and starting the Hadoop daemons.
5. Running a sample word count MapReduce job to count words in a file and viewing the output.
The document provides steps to install Hadoop on a single node and run a sample MapReduce job. The key steps are:
1. Downloading and extracting the Hadoop tar file.
2. Configuring passwordless SSH and setting environment variables like JAVA_HOME.
3. Editing configuration files like core-site.xml and hdfs-site.xml.
4. Formatting the namenode and starting the Hadoop daemons.
5. Running a sample word count MapReduce job to count words in a file and viewing the output.
Download as ODT, PDF, TXT or read online from Scribd
Download as odt, pdf, or txt
You are on page 1/ 5
Hadoop Installation
Copy below installation file from http://www-eu.apache.org/dist/hadoop/common/hadoop-2.6.5/
hadoop-2.6.5.tar.gz 1. to home folder ( Login is same as India domain login) .
2. Extract tar file
tar -xvf hadoop-2.6.5.tar.gz
3. Generate Key
• [xxxxxx@01hw744052 hadoop-2.6.5]$ ssh-keygen
Generating public/private rsa key pair. Enter file in which to save the key (/home/xxxxxx/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/xxxxxx/.ssh/id_rsa. Your public key has been saved in /home/xxxxxx/.ssh/id_rsa.pub. The key fingerprint is: ee:d9:c8:a7:80:cc:6b:d8:d7:c6:47:b6:62:b2:00:ad xxxxxx@01hw744052.tcs-mobility.com The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | . | | .. S | | =.. o | | Eo= .o.o . | | . ooo+*++ | | ....=*=. | +-----------------+ 1. Setup passwordless ssh [xxxxxx@01hw744052 hadoop-2.6.5]$ ssh-copy-id -i empid@localhost The authenticity of host 'localhost (<no hostip for proxy command>)' can't be established. RSA key fingerprint is 04:f4:d1:1c:62:ad:1f:29:71:5c:50:bd:93:49:a9:68. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (RSA) to the list of known hosts. xxxxxx@localhost's password: Now try logging into the machine, with "ssh 'xxxxxx@localhost'", and check in:
.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. 2. Verify passwordless ssh
7. Edit the core-site.xml( in /etc/hadoop) file as follows
<property> <name>fs.default.name</name> <value>hdfs://127.0.0.1:9000</value> </property> <property> <name>fs.tmp.dir</name> <value>/home/<empid>/hadoop-2.6.5/tmp</value> </property> NB: Make sure directory configured as fs.tmp.dir are created.
8. Edit the hdfs-site.xml ( in /etc/hadoop)
//Update the path to reflect your installation home directory
<property> <name>dfs.name.dir</name> <value>/home/<empid>/hadoop-2.6.5/dfs/nn</value> </property> <property> <name>dfs.data.dir</name> <value>/home/<empid>/hadoop-2.6.5/dfs/dn</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> NB: Make sure the directories configured as dfs.name.dir and dfs.data.dir are created.
15/05/22 12:36:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Leo 4 hi 2
19. To copy files from HDFS onto local system, use below command. (Optional) hadoop fs -copyToLocal /output1/part-r-00000 .
