How to deal with executor memory and driver memory in Spark?

Question

I am confused about dealing with executor memory and driver memory in Spark.

My environment settings are as below:

• Memory 128 G, 16 CPU for 9 VM
• Centos
• Hadoop 2.5.0-cdh5.2.0
• Spark 1.1.0

Input data information:

• 3.5 GB data file from HDFS

For simple development, I executed my Python code in standalone cluster mode (8 workers, 20 cores, 45.3 G memory) with spark-submit. Now I would like to set executor memory or driver memory for performance tuning.

From the Spark documentation, the definition for executor memory is

Amount of memory to use per executor process, in the same format as JVM memory strings (e.g. 512m, 2g).

How about driver memory?

Answer 1

Executors are worker nodes' processes in charge of running individual tasks in a given Spark job and The spark driver is the program that declares the transformations and actions on RDDs of data and submits such requests to the master.

Now, talking about driver memory, the amount of memory that a driver requires depends upon the job to be executed.

In Spark, the executor-memory flag controls the executor heap size (similarly for YARN and Slurm), the default value is 512MB per executor. And the driver-memory flag controls the amount of memory to allocate for a driver, which is 1GB by default and should be increased in case you call a collect() or take(N) action on a large RDD inside your application.

Spark shell required memory = (Driver Memory + 384 MB) + (Number of executors * (Executor memory + 384 MB))

Here 384 MB is maximum memory (overhead) value that may be utilized by Spark when executing jobs.

Learn Spark with this Spark Certification Course by Intellipaat.

Answer 2

According to your configuration, do the following to dial up executor memory and driver memory for your Spark job against a standalone cluster:

A. Executor vs. Driver Memory

Executor memory : This refers to the portion of the memory assigned to each executor, or simply put, the worker process that executes the actual process on data. Executors would need adequate memory to run their part of the data, complete tasks, and cache when necessary.

Driver Memory This is the space allocated to the driver process, coordinating the execution of tasks in the cluster. Driver memory is essential because it's used for such things as running Spark control flow, aggregation of output especially when using collect() and job metadata management; hence it should have quite enough memory to accommodate lots of communications with all those executors and large-sized operations.

B. Draft Suggestions for Preliminary Modifications

Executor Memory: You're running a local cluster with 8 workers and 45.3 GB total memory. For the sake of simplicity, assume approximately 5 GB of memory per worker-that is, (45.3 GB divided by 9 minus the driver's share). You can start with setting executor memory to about 5 GB:

--conf spark.executor.memory=5g

Driver Memory: The driver should take lesser memory as compared to executors. As driver is coordinator process, 2 GB for driver memory should be around.

--conf spark.driver.memory=2g

C. Calibration and Tracking

Driver and Executor Memory Overhead: The overhead memory is required for avoiding OOM errors. One should allocate some additional overhead memory to avoid OOM error. This can be achieved by the following configuration

--conf spark.executor.memoryOverhead=512m

--conf spark.driver.memoryOverhead=512m

Monitor Spark UI: The Spark UI (http://<driver-node>:4040) will give you information about the memory usage, efficiency of the executors, and areas of bottlenecks.If much garbage collection or failure in the executor is seen, then increase some memory and overhead in the executor.

D. Sample spark-submit command

With such configuration, your command would look something like this:.

-submit \

--master spark:// <your-master-url>:7077

--conf spark.executor.memory=5g \

--conf spark.driver.memory=2g \

--conf spark.executor.memoryOverhead=512m \

--conf spark.driver.memoryOverhead=512m \

your_script.py

E. Testing and Calibration by Job Difficulty The jobs don't work. Increase the memory settings in 512 MB to determine the stable configuration. The jobs are running fine but could be faster. Increase executor memory and see if that improves anything.