Write a file in hdfs with Java

Question

I want to create a file in HDFS and write data in that. I used this code:

Configuration config = new Configuration();    
FileSystem fs = FileSystem.get(config);
Path filenamePath = new Path("input.txt"); 
try {
    if (fs.exists(filenamePath)) {
        fs.delete(filenamePath, true);
    }

    FSDataOutputStream fin = fs.create(filenamePath);
    fin.writeUTF("hello");
    fin.close();
}

It creates the file, but it doesn't write anything in it. I searched a lot but didn't find anything. What is my problem? Do I need any permission to write in HDFS?

Answer 1

Whenever you want to write a file in HDFS, always pass the URI(a string of characters used to identify a resource on a computer network, of which the best-known type is the web address or URL.) while getting the filesystem.

Configuration configuration = new Configuration();

FileSystem hdfs = FileSystem.get( new URI( "hdfs://localhost:50070" ), configuration );

Path file = new Path("hdfs://localhost:50070/s2019/batch/tab.html");

if ( hdfs.exists( file )) { hdfs.delete( file, true ); }

OutputStream os = hdfs.create( file,

   new Progressable() {

       public void progress() {

           out.println("...bytes written: [ "+bytesWritten+" ]");

       } });

BufferedWriter bw = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );

bw.write("Hello World");

bw.close();

hdfs.close();

Another approach to solve your problem:

You can just add the following lines to your code:

config.addResource(new Path("/HADOOP_HOME/conf/core-site.xml"));

config.addResource(new Path("/HADOOP_HOME/conf/hdfs-site.xml"));

OR just go to your Hadoop Configuration folder and define the HADOOP_CONF_DIR environment variable there.

If you have any doubt regarding the Hadoop, you can refer the following video: