The way to check a HDFS directory's size? Ask

Question

How can I find the size of a directory, using Hadoop?

Answer 1

You can use the “hadoop fs -ls command”.

This command displays the list of files in the current directory and all it’s details.In the output of this command, the 5th column displays the size of file in bytes.

For e.g.

command hadoop fs -ls     

input gives following output:

Found 1 items

-rw-r--r--   1 hduser supergroup  36789 2012-07-19 20:57 /user/hduser/input/shivangi

The size of file shivangi is 36789 bytes.

Answer 2

hadoop fs -du -s -h /path/to/dir

The above displays a directory's size in a readable form.

Do check out this tutorial video if you want to learn from scratch:

Answer 3

To find the size of a directory in HDFS with Hadoop, try these commands:

Basic Disk Usage with -du:

hadoop fs -du /path/to/directory

This lists the size of each file and folder inside the specified directory.

Get a Total Size Summary with -du -s:

hadoop fs -du -s /path/to/directory

Adding -s gives you the combined size of everything in the directory, instead of individual files.

Check Available Space with -df -h:

hadoop fs -df -h /path/to/directory

This displays the used and available storage in a readable format.

These commands are handy for checking how much HDFS storage you’re using