Apache Spark does not delete temporary directories

Question

After a spark program completes, there are 3 temporary directories remain in the temp directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb7

The directories are empty.

And when the Spark program runs on Windows, a snappy DLL file also remains in the temp directory. The file name is like this: snappy-1.0.4.1-6e117df4-97b6-4d69-bf9d-71c4a627940c-snappyjava

They are created every time the Spark program runs. So the number of files and directories keeps growing.

How can let them be deleted?

Spark version is 1.3.1 with Hadoop 2.6.

Answer 1

Using the "local" mode for testing purposes, you can solve this issue by creating a custom temp folder before running a test and then delete it manually (in my case I used local mode in JUnit in order to delete the temp folder automatically).

You should change the path to the temp folder for Spark by spark.local.dir property.

SparkConf conf = new SparkConf().setMaster("local")

                                .setAppName("test")

                                .set("spark.local.dir", "/tmp/spark-temp");

After the test is completed delete the /tmp/spark-temp folder manually.