0 votes
1 view
in BI by (13.4k points)

I am trying to make extract information from Spark SQL. Following error message showing while creating extract.

[Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 906 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)'.

1 Answer

0 votes
by (36.7k points)

A quick fix is just changing the setting in your execution context.

spark.sql("set spark.driver.maxResultSize = 8G")

...