How can I debug spark application locally?

Question

I am very new to spark and I just would like to learn step by step how to debug a spark application locally? Could anyone please detail the steps needed to do this?

I can run the simpleApp on the spark website locally from the command line but I just need to step through the code and see how it works. Any help is very much appreciated.

Answer 1

Fire up the Spark shell. This is straight from the Spark documentation:

./bin/spark-shell --master local[2]

You will also see the Spark shell referred to as the REPL. It is by far the best way to learn Spark. I spend 80% of my time in the Spark shell and the other 20% translating the code into my application.

Note:  if the application is written in Java then the Scala spark-shell can load Java jars, then you can run your app and or methods from Scala.
Also, if you are using IntelliJ IDE, I would suggest you follow this tutorial.

If you want to know more about Spark, then do check out this awesome video tutorial: