How to run a script in PySpark

Question

I'm trying to run a script in the pyspark environment but so far I haven't been able to. How can I run a script like python script.py but in pyspark?

Answer 1

For PySpark 2.0+, just execute script file in environment variable PYTHONSTARTUP, so you can run:

PYTHONSTARTUP=code.py pyspark

Also, note that running python applications through pyspark is not supported as of Spark 2.0.

So in that case you can do this:

./bin/spark-submit mypythonfile.py