How to set hadoop configuration values from pyspark

Question

The Scala version of SparkContext has the property

sc.hadoopConfiguration

I have successfully used that to set Hadoop properties (in Scala)

e.g.

sc.hadoopConfiguration.set("my.mapreduce.setting","someVal")

However the python version of SparkContext lacks that accessor. Is there any way to set Hadoop configuration values into the Hadoop Configuration used by the PySpark context?

Answer 1

As in Scala you used :

sc.hadoopConfiguration.set("my.mapreduce.setting","someVal")

Similarly to set Hadoop configuration values into the Hadoop Configuration used by the PySpark context, do:

sc._jsc.hadoopConfiguration().set('my.mapreduce.setting', 'someVal')