0 votes
1 view
in BI by (17.6k points)

I am trying to make extract information from Spark SQL. Following error message showing while creating extract.

[Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 906 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)'.

1 Answer

0 votes
by (47.2k points)

A quick fix is just changing the setting in your execution context.

spark.sql("set spark.driver.maxResultSize = 8G")

Welcome to Intellipaat Community. Get your technical queries answered by top developers !