HDFS is the primary or major component of the Hadoop ecosystem which is responsible for storing large data sets of structured or unstructured data across various nodes and thereby maintaining the metadata in the form of log files. To use the HDFS commands, first we need to start the Hadoop services using the following command: sbin/start-all.sh Hadoop HDFS Commands With the help of the HDFS command, we can perform Hadoop HDFS file operations like changing the file permissions, viewing the file contents, creating files or directories, copying file/directory from the local file system to HDFS or vice-versa, etc. Before starting with the HDFS command, we have to start the Hadoop services. To start the Hadoop services do the following: 1. Move to the ~/hadoop-3.1.2 directory 2. Start Hadoop service by using the command sbin/start-dfs.sh In this Hadoop Commands tutorial, we have mentioned the top 10 Hadoop HDFS commands with their usage, examples, and description. Let us now start with the HDFS commands.
1. version Hadoop HDFS version Command Usage: version Hadoop HDFS version Command Example: Before working with HDFS you need to Deploy Hadoop, follow this guide to Install and configure Hadoop 3. hadoop version
Hadoop HDFS version Command Description:
The Hadoop fs shell command version prints the Hadoop version. 2. mkdir Hadoop HDFS mkdir Command Usage: hadoop fs –mkdir /path/directory_name Hadoop HDFS mkdir Command Example 1: In this example, we are trying to create a newDataFlair named directory in HDFS using the mkdir command.
Using the ls command, we can check for the directories in HDFS.
Example 2:
Hadoop HDFS mkdir Command Description:
This command creates the directory in HDFS if it does not already exist. Note: If the directory already exists in HDFS, then we will get an error message that file already exists. Use hadoop fs mkdir -p /path/directoryname, so not to fail even if directory exists. Learn various features of Hadoop HDFS from this HDFS features guide. 3. ls Hadoop HDFS ls Command Usage: hadoop fs -ls /path Hadoop HDFS ls Command Example 1: Here in the below example, we are using the ls command to enlist the files and directories present in HDFS.
Hadoop HDFS ls Command Description:
The Hadoop fs shell command ls displays a list of the contents of a directory specified in the path provided by the user. It shows the name, permissions, owner, size, and modification date for each file or directories in the specified directory. Hadoop HDFS ls Command Example 2:
Hadoop HDFS ls Description:
This Hadoop fs command behaves like -ls, but recursively displays entries in all subdirectories of a path. 4. put Hadoop HDFS put Command Usage: haoop fs -put <localsrc> <dest> Hadoop HDFS put Command Example: Here in this example, we are trying to copy localfile1 of the local file system to the Hadoop filesystem.
Hadoop HDFS put Command Description:
The Hadoop fs shell command put is similar to the copyFromLocal, which copies files or directory from the local filesystem to the destination in the Hadoop filesystem. 5. copyFromLocal Hadoop HDFS copyFromLocal Command Usage: hadoop fs -copyFromLocal <localsrc> <hdfs destination> Hadoop HDFS copyFromLocal Command Example: Here in the below example, we are trying to copy the ‘test1’ file present in the local file system to the newDataFlair directory of Hadoop.
Hadoop HDFS copyFromLocal Command Description:
This command copies the file from the local file system to HDFS. Learn Internals of HDFS Data Read Operation, How Data flows in HDFS while reading the file. Any Doubt yet in Hadoop HDFS Commands? Please Comment. 6. get Hadoop HDFS get Command Usage: hadoop fs -get <src> <localdest> Hadoop HDFS get Command Example: In this example, we are trying to copy the ‘testfile’ of the hadoop filesystem to the local file system. Hadoop HDFS get Command Description: The Hadoop fs shell command get copies the file or directory from the Hadoop file system to the local file system.
Learn: Rack Awareness, High Availability
7. copyToLocal Hadoop HDFS copyToLocal Command Usage: hadoop fs -copyToLocal <hdfs source> <localdst> Hadoop HDFS copyToLocal Command Example: Here in this example, we are trying to copy the ‘sample’ file present in the newDataFlair directory of HDFS to the local file system.
We can cross-check whether the file is copied or not using the ls command.
Hadoop HDFS copyToLocal Description:
copyToLocal command copies the file from HDFS to the local file system. 8. cat Hadoop HDFS cat Command Usage: hadoop fs –cat /path_to_file_in_hdfs Hadoop HDFS cat Command Example: Here in this example, we are using the cat command to display the content of the ‘sample’ file present in newDataFlair directory of HDFS.
Hadoop HDFS cat Command Description:
The cat command reads the file in HDFS and displays the content of the file on console or stdout. 9. mv Hadoop HDFS mv Command Usage: hadoop fs -mv <src> <dest> Hadoop HDFS mv Command Example: In this example, we have a directory ‘DR1’ in HDFS. We are using mv command to move the DR1 directory to the DataFlair directory in HDFS.
Hadoop HDFS mv Command Description:
The HDFS mv command moves the files or directories from the source to a destination within HDFS. 10. cp Hadoop HDFS cp Command Usage: hadoop fs -cp <src> <dest> Hadoop HDFS cp Command Example: In the below example we are copying the ‘file1’ present in newDataFlair directory in HDFS to the dataflair directory of HDFS.
Hadoop HDFS cp Command Description:
The cp command copies a file from one directory to another directory within the HDFS. So this was all on Hadoop HDFS Commands. Hope you like it.
