Why does ./bin/spark-shell give WARN NativeCodeLoader: Unable to load native-hadoop library for your platform?

Question

On Mac OS X, I compiled Spark from the sources using the following command:

abc:~/efg/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SPARK_GANGLIA_LGPL=true xsbt
...

[info] Set current project to root (in build file:/Users/abc/efg/spark/)
> ; clean ; assembly
...
[info] Packaging /Users/abc/efg/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar ...
[info] Done packaging.
[info] Done packaging.
[success] Total time: 1964 s, completed May 9, 2014 5:07:45 AM

When I started ./bin/spark-shell I noticed the following WARN message:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

What might be the issue?

Answer 1

The native hadoop library does not work with Cygwin or the Mac OS X platform. It is supported on *nix platforms only. 

It is mainly used on the GNU/Linus platform 

The WARN message that you are getting should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.