what's the difference between “hadoop fs” shell commands and “hdfs dfs” shell commands?

Question

Are they same?

In my view "hadoop fs" command shows the hdfs files and the "hdfs dfs" command shows the local files, why is that?

Answer 1

The “fs” term refers to a generic file system, which by the definition can point to ANY file system ( including HDFS), but dfs is very specific. On the other hand, “DFS” refers precisely to Hadoop Distributed File System access. So when we use FS it can perform operation related to local or hadoop distributed file system and dfs can perform operation related to hadoop distributed file system only.

So,

1. hadoop fs <args>

It is used when we are dealing with different file systems such as Local FS,  HDFS etc.

2. hdfs dfs <args>

It is used when we are dealing for operations related to HDFS.

Another command, which looks similiar to these two is

3. hadoop dfs

This command should not be used, as it is deprecated. Even if you use it, it will send the command to hdfs dfs.

Comments

Thank you for the explanation.

Answer 2

hadoop fs <args>

fs is used for a generic file system and it can point to any file system such as a local file system, HDFS, WebHDFS, S3 FS, etc.

hadoop dfs <args>

hdfs dfs <args>

dfs points to the Distributed File System and it is specific to HDFS. You can use it to execute operations on HDFS. Now it is deprecated, and you have to use hdfs dfs instead of hadoop dfs.

Hope this answer helps you!

Comments

It worked for me.

Answer 3

First of all, I’m fairly certain that the commands are case-sensitive and they both should be lowercased: hdfs dfs and hadoop fs.

That said, let me direct you to the official documentation. To be simple, hadoop fs is more “generic” command that allows you to interact with multiple file systems including Hadoop, whereas hdfs dfs is the command that is specific to HDFS.

Note that hdfs dfs and hadoop fs commands become synonymous if the filing system which is used is HDFS. In fact, if you issue hadoop fs command, it will tell you that it has been deprecated and you should use hdfs dfs instead.

Comments

Nice explanation. Couldn't find a better explanation on the web.

---

I think hdfs dfs -ls file:/// seems to work just as well

Answer 4

Following are the three commands which appear the same but have minute differences

1. hadoop fs {args}
2. hadoop dfs {args}
3. hdfs dfs {args}

  hadoop fs <args>

FS relates to a generic file system which can point to any file systems like local, HDFS, etc. So this can be used when you are dealing with different file systems such as Local FS, (S)FTP, S3, and others.

 hadoop dfs <args>

dfs is very specific to HDFS. would work for operation relates to HDFS. This has been deprecated and we should use hdfs dfs instead.

 hdfs dfs <args>

same as 2nd i.e would work for all the operations related to HDFS and is the recommended command instead of hadoop dfs

Below is the list categorized as hdfs commands:

namenode|secondarynamenode|datanode|dfs|dfsadmin|fsck|balancer|fetchdt|oiv|dfsgroups

So even if you use hadoop dfs, it will look to locate hdfs and delegate that command to hdfs dfs.

You can also refer to the following video regarding Hadoop, as it will clear all your doubts regarding the same: 

Comments

Very well explained!

Answer 5

Here, fs refers to any file system, it can be local or HDFS but dfs refers to only the HDFS file system.

Therefore, if you need to perform access/transfer data between different filesystem, fs is the best command to use.

Comments

Thankyou ! well explained , i got it

Answer 6

The difference between “hadoop fs” shell commands and “hdfs dfs” shell commands is as follows:-

The term fs refers to any file system and it could be local or HDFS. Where as the term dfs refers to only HDFS file system. So if you need to perform access/transfer data between different filesystem, fs is the way to go.