Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (19k points)

I am trying to change the default configuration of Spark Session. But it is not working.

spark_session  = SparkSession.builder




spark_session.conf.set("spark.executor.memory", '8g')

spark_session.conf.set('spark.executor.cores', '3')

spark_session.conf.set('spark.cores.max', '3')


sc = spark_session.sparkContext

But if I put the configuration in Spark submit, then it works fine for me.

spark-submit --master ip --executor-cores=3 --diver 8G

1 Answer

0 votes
by (33.1k points)

Simply open PySpark shell and check the settings:


Now you can execute the code and again check the setting of the Pyspark shell.

You first have to create conf and then you can create the Spark Context using that configuration object.

config = pyspark.SparkConf().setAll([('spark.executor.memory', '8g'), ('spark.executor.cores', '3'), ('spark.cores.max', '3'), ('spark.driver.memory','8g')])


sc = pyspark.SparkContext(conf=config)

I hope this answer helps you!

Related questions

Browse Categories