Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (11.4k points)

The Scala version of SparkContext has the property

sc.hadoopConfiguration


I have successfully used that to set Hadoop properties (in Scala)

e.g.

sc.hadoopConfiguration.set("my.mapreduce.setting","someVal")


However the python version of SparkContext lacks that accessor. Is there any way to set Hadoop configuration values into the Hadoop Configuration used by the PySpark context?

1 Answer

0 votes
by (32.3k points)

As in Scala you used :

sc.hadoopConfiguration.set("my.mapreduce.setting","someVal")

Similarly to set Hadoop configuration values into the Hadoop Configuration used by the PySpark context, do:

sc._jsc.hadoopConfiguration().set('my.mapreduce.setting', 'someVal')

Browse Categories

...