Spark cluster full of heartbeat timeouts, executors exiting on their own

Question

My Apache Spark cluster is running an application that is giving me lots of executor timeouts:

10:23:30,761 ERROR ~ Lost executor 5 on slave2.cluster: Executor heartbeat timed out after 177005 ms
10:23:30,806 ERROR ~ Lost executor 1 on slave4.cluster: Executor heartbeat timed out after 176991 ms
10:23:30,812 ERROR ~ Lost executor 4 on slave6.cluster: Executor heartbeat timed out after 176981 ms
10:23:30,816 ERROR ~ Lost executor 6 on slave3.cluster: Executor heartbeat timed out after 176984 ms
10:23:30,820 ERROR ~ Lost executor 0 on slave5.cluster: Executor heartbeat timed out after 177004 ms
10:23:30,835 ERROR ~ Lost executor 3 on slave7.cluster: Executor heartbeat timed out after 176982 ms

However, in my configuration I can confirm I successfully increased the executor heartbeat interval: enter image description here

When I visit the logs of executors marked as EXITED (i.e.: the driver removed them when it couldn't get a heartbeat), it appears that executors killed themselves because they didn't receive any tasks from the driver:

16/05/16 10:11:26 ERROR TransportChannelHandler: Connection to /10.0.0.4:35328 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong.
16/05/16 10:11:26 ERROR CoarseGrainedExecutorBackend: Cannot register with driver: spark://[email protected]:35328

How can I turn off heartbeats and/or prevent the executors from timing out?

Answer 1

Missing heartbeats or executors being killed such problems usually occur due to OOMs.

I would suggest you inspect the logs on the individual executors (look for the text "running beyond physical memory"). If you have got some no. of executors and you find it cumbersome to inspect all of the logs manually, I recommend you to monitor your job in the Spark UI while it runs. As soon as a task fails, it will report the cause in the UI, so it's easy to see. Also, keep in mind that some tasks will report failure due to missing executors that have already been killed, so make sure that you are looking at causes for each of the individual failing tasks.

Most of the OOM problems can be solved quickly by simply repartitioning your data at appropriate places in your code (again look at the Spark UI for hints as to where there might be a need for a call to repartition). Repartitioning can be your key in many such cases. 

You can also scale up your machines to accommodate the need for memory.