0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I'm trying to get the path to spark.worker.dir for the current sparkcontext.

1 Answer

0 votes
by (32.5k points)

Yes, I would suggest you to try the method below in order to get the current spark context settings.

SparkConf.getAll()

as accessed by

SparkContext.sc._conf

Another method specifically for Spark 2.1+

spark.sparkContext.getConf().getAll() 

where spark is your sparksession (gives you a dict with all configured settings)

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...