Apache Hadoop Commands

List of important Apache Hadoop Commands

The basic commands of Hadoop are listed and explained in this Commands tutorial. There are many more commands in “$Hadoop_HOME/bin/Hadoop fs” but these basic operations will get you started. Running ./bin/hadoop dfs with no additional arguments will list all the commands that can be run with the Fs Shell system. In addition you have a help option if you got stuck. $HADOOP_HOME/bin/hadoop fs –help command Name will display a short description and a usage summary for the operation in question.

The following are the conventions used for parameters:

“<path>”                                       – It defines the name of any directory or file.

“<path>…”                                    – It denotes one or more directory or file names.

“<file>”                                         – Refers to any file name.

“<src>” and “<dest>”                    – They refer to path names in a directed operation.

“<localsrc>” and “<localdest>”       –  same as mentioned above they are paths but on the local file                   system.

 

Command Usage: Version

Command Example: 

hdfs dfs version

Description: 

It prints the Hadoop version.

 

Command Usage: ls <path>

Command Example: 

hdfs dfs – ls /user/beyondcorner/dir1

Description:

This command will give you a list of files or sub-files inside that particular directory. It also give details about the file size, file credentials  and date of modification.

 

Command Usage: lsr <path>

Command Example:

hdfs dfs – lsr /user/beyondcorner/dir1

Description:

It behaves like -ls, displays entries in all subdirectories of the path but it does that recursively.

 

Command Usage:mkdir <path>

Command Example:

hdfs dfs – mkdir /user/beyondcorner/dir1

Description:

 It creates a directory by taking path URI’s as an argument.

It creates any missing parent directories in the path (e.g., mkdir -p in Linux).

 

put :   Command Usage:

put <localSrc> <dest>

Command Example:

hdfs dfs -put /home/beyondcorner/Desktop/sample /user/beyondcorner/dir1

Description:

It copies the file or directory from the local file system to the destination within the HDFS.

 

copyFromLocal : Command Usage:

copyFromLocal <localSrc> <dest>

Command Example:

hdfs dfs -copyFromLocal /home/beyondcorner/Desktop/sample /user/beyondcorner/dir1

Description:

It copies the file or directory from the given local file reference to the destination within the HDFS.

 

get :  Command Usage:

get [-crc] <src> <localDest>

Command Example:

hdfs dfs -get /user/beyondcorner/dir2/sample /home/beyondcorner/Desktop

Description:

 It copies the file or directory in HDFS identified by the source to the given local file destination identified by the localDest.

 

CopyToLocal : Command Usage:

copyToLocal <src> <localDest>

Command Example:

hdfs dfs -get /user/beyondcorner/dir2/sample /home/beyondcorner/Desktop

Description:

The source  given identifies the file or directory in HDFS then it is copied to the given local file reference.

 

cat: Command Usage:

cat <file-name>

Command Example:

hdfs dfs -cat /user/beyondcorner/dir1/sample

Description:

This command displays the contents of the filename on console or stdout.

 

mv : Command Usage:

mv <src> <dest>

Command Example:

hadoop fs -mv /user/beyondcorner/dir1/purchases.txt /user/beyondcorner/dir2

Description:

It moves the file or directory indicated by the source to destination, within HDFS.

 

cp  : Command Usage:

cp <src> <dest>

Command Example:

hadoop fs -cp /user/beyondcorner/dir2/purchases.txt /user/beyondcorner/dir1

Description:

The source identifies the file or directory and it is copied to destination, within HDFS.

 

moveFromLocal  : Command Usage:

moveFromLocal <localSrc> <dest>

Command Example:

hdfs dfs -moveFromLocal /home/beyondcorner/Desktop/sample /user/beyondcorner/dir1

Description:

Copies the file or directory from the given local source file system to the given destination within HDFS and on copying successfully system deletes the local copy.

 

moveToLocal  : Command Usage:

moveToLocal <src> <localDest>

Command Example:

hdfs dfs -moveToLocal /user/beyondcorner/dir2/sample /user/beyondcorner/Desktop

Description:

It copies the file or directory in HDFS identified by the given source to the given local file destination and on copying successfully system deletes the HDFS copy.

 

tail  : Command Usage:

hdfs dfs -tail [-f] <filename>

Command Example:

hdfs dfs -tail -f /user/beyondcorner/dir2/temp.txt

Description:

It shows last 1KB data of mentioned source file in console.

 

rm  : Command Usage:

 rm <path>

 Command Example:

hdfs dfs -rm /user/beyondcorner/dir2/sample 

Description:

It removes the file or empty the directory present on the path given by the user.

 

expunge  : Command Usage:

 hdfs dfs -expunge

 Command Example:

hdfs dfs -expunge

Description:

Used to empty the trash.

 

chown  : Command Usage:

 hdfs dfs -chown [-R] [OWNER] [:[GROUP]] <path>…

 Command Example:

hdfs dfs -chown -R beyondcorner /opt/hadoop/logs

Description:

It changes the owner of files or directories which are identified by the path. With -R, changes are made recursively.

 

chgrp  : Command Usage:

 hdfs dfs -chgrp [-R] <NewGroupName> <file or directory name>

 Command Example:

hdfs dfs -chgrp [-R] New Group sample

Description:

Hadoop chgrp shell command is used to set the owing group for files or directories identified by path. If  -R is specified changes can be made  recursively.

 

setrep  : Command Usage:

hdfs dfs setrep [-R] [-w] rep <path>

 Command Example:

hdfs dfs -setrep -w 3 /user/beyondcorner/dir1

Description:

This command changes the replication factor of a file or directory. If the path entered referes to a directory then this command changes the replication factor for all files present inside that directory.

Options:
The -w flag is to request the command to wait for the replication process to get completed.

The -R flag does not make any changes and it is accepted for backward compatibility.

 

du  : Command Usage:

 hdfs dfs -du <path>

 Command Example:

hdfs dfs -du /user/beyondcorner/dir1/temp

Description:

Du stands for disk usage so it shows the disk usage in the capacity of bytes, for all the files present on the path provided by the user.

 

df  : Command Usage:

hdfs dfs – df [-h] URI [URI …]

 Command Example:

hdfs dfs -df -h

Description:

It is used to display free space available.

 

touchz  : Command Usage:

 hdfs dfs -touchz <path>

 Command Example:

hdfs dfs -touchz /user/beyondcorner/dir2

Description:

It is used to create file with current time as a timestamp in the given file path. If the given path had a non-zero size file already the operation fails.

 

test  : Command Usage:

hdfs dfs -test -[ezd] URI

 Command Example:

hdfs dfs -test -e sample

hdfs dfs -test -z sample

hdfs dfs -test -d sample

Description

The test command is used for file test operations.

Output will be 1 if the given file path exists, else 0.
Options:
-d: if the path given by the user is a directory, then it gives 0 output.
-e: if the path given by the user exists, then it gives 0 output.
-f: if the path given by the user is a file, then it gives 0 output.
-s: it gives 0 as output, if the path given by the user is empty.
-z: it gives 0 output, if the file is zero length.

 

text  : Command Usage:

 hdfs dfs -text <source>

 Command Example:

hdfs dfs -text /user/beyondcorner/dir1/temp

Description:

It is used to format the output file as same as the given source file.

 

stat  : Command Usage:

hdfs dfs -stat <path>

 Command Example:

hdfs dfs -stat /user/beyondcorner/dir1

Description:

It is used to print the information about the path.

%b: If the format is a string which accepts file size in blocks.

%n: Filename

%o: Block size

%r: replication

%y, %Y: modification date

 

chmod  : Command Usage:

 hdfs dfs -chmod [-R] mode,mode,… <path>…

 Command Example:

hdfs dfs -chmod 777 /user/beyondcorner/dir1/sample 

Description:

It is used to change the permissions or provide access to the files. With -R, the changes are made repeatedly down through the directory structure. The owner of the file must use this command to give permissions.

 

appenfToFile  : Command Usage:

 hdfs dfs –appendToFile <localSource>… <dest>

 Command Example:

hdfs dfs -appendToFile /home/beyondcorner/Desktop/sample /user/beyondcorner/dir1

Description:

It is used to append single sources or multiple sources from local file system to the file system at the destination. It also reads input from standard input and adds it to destination file system.

 

checksum  : Command Usage:

 hdfs dfs –checksum URI

 Command Example:

hdfs dfs -checksum /user/beyondcorner/dir1/sample

Description:

It is used to return the checksum information of the given file.

 

count  : Command Usage:

 hdfs dfs –count [-q] <paths>

 Command Example:

hdfs dfs -count /user/beyondcorner 

Description:

It is used to counts the number of directories, number of files present and bytes under the given path.

 

find  : Command Usage:

hadoop fs -find <path> … <expression> …

 Command Example:

hadoop fs -find /user/beyondcorner/dir1/ -name sample -print 

Description:

It is used to find all files that match the specified expression and performs all the actions to them which are selected. If no path is specified in the command then defaults to the present working directory. If none of the expression is specified in the command then defaults to -print.

 

help  : Command Usage:

hadoop fs –help

 Command Example:

hadoop fs -help 

Description:

It is used to displays usage information for the commands entered by the user.

 

truncate  : Command Usage:

hadoop fs –truncate [-w] <length> <paths>

 Command Example:

hadoop fs -help 

Description:

Used to truncate all the files with a specified length that matches the specified file pattern.
Options:
Block recovery takes place and the -w flag requests that if necessary the command waits for block recovery to get completed. If there is no  -w flag then the file may remain open for a period of time the recovery process is going on. At this time reopening the file for append is not possible.

 

usage  : Command Usage:

hadoop fs –usage command

 Command Example:

hadoop fs -usage mkdir 

Description:

It is used to displays the help for an individual command.

Its all about the Apache Hadoop Commands, Let us know if you having any doubt.