Unable to Create Extract - Tableau and Spark SQL

Question

I am trying to make extract information from Spark SQL. Following error message showing while creating extract.

[Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 906 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)'.

Answer 1

A quick fix is just changing the setting in your execution context.

spark.sql("set spark.driver.maxResultSize = 8G")