0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)
I'm trying to run a script in the pyspark environment but so far I haven't been able to. How can I run a script like python script.py but in pyspark?

1 Answer

0 votes
by (32.5k points)

For PySpark 2.0+, just execute script file in environment variable PYTHONSTARTUP, so you can run:

PYTHONSTARTUP=code.py pyspark

Also, note that running python applications through pyspark is not supported as of Spark 2.0.

So in that case you can do this:

./bin/spark-submit mypythonfile.py

Related questions

Welcome to Intellipaat Community. Get your technical queries answered by top developers !