Why does spark-submit and spark-shell fail with “Failed to find Spark assembly JAR. You need to build Spark before running this program”?

Question

I was trying to run spark-submit and I get "Failed to find Spark assembly JAR. You need to build Spark before running this program." When I try to run spark-shell I get the same error. What I have to do in this situation.

Answer 1

Your Spark package doesn't include compiled Spark code. That's why you are getting such error message when you are using these scripts spark-submit and spark-shell.

You have to download one of pre-built version in "Choose a package type" section from the Spark download page.

For Windows:

If you have done your Spark installation in a directory that contains a space in its path e.g. D:\ABC Folder\Spark, you will encounter errors. So, in that scenario I will suggest you to move it to the root or another directory with no spaces.