Apache Spark: How to use pyspark with Python 3

Question

I built Spark 1.4 from the GH development master, and the build went through fine. But when I do a bin/pyspark I get the Python 2.7.9 version. How can I change this?

Answer 1

You need to set the environment variable first then execute /bin/pyspark.

• export PYSPARK_PYTHON=python3
• ./bin/pyspark  

If you want to know more about Spark, then do check out this awesome video tutorial: