Intellipaat Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (11.4k points)

I'm a dummy on Ubuntu 16.04, desperately attempting to make Spark work. I've tried to fix my problem using the answers found here on stackoverflow but I couldn't resolve anything. Launching spark with the command ./spark-shell from bin folder I get this message

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable".
 

I'm using Java version is

java version "1.8.0_101
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode).


Spark is the latest version: 2.0.1 with Hadoop 2. 7. I've also retried with an older package of Spark, the 1.6.2 with Hadoop 2.4 but I get the same result. I also tried to install Spark on Windows but it seems harder than doing it on Ubuntu.

1 Answer

0 votes
by (32.3k points)

I think maybe it's the Hadoop library which is compiled for 32bits, while you are on 64bit.

But I would suggest you to fix the following steps, after going through your steps again:

 

  • download Hadoop binaries

  • unpack to directory of your choice

  • set HADOOP_HOME to point to that directory.

  • add $HADOOP_HOME/lib/native to LD_LIBRARY_PATH.

31k questions

32.8k answers

501 comments

693 users

Browse Categories

...