Back

Explore Courses Blog Tutorials Interview Questions
0 votes
1 view
in Big Data Hadoop & Spark by (11.4k points)

After a spark program completes, there are 3 temporary directories remain in the temp directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb7

The directories are empty.

And when the Spark program runs on Windows, a snappy DLL file also remains in the temp directory. The file name is like this: snappy-1.0.4.1-6e117df4-97b6-4d69-bf9d-71c4a627940c-snappyjava

They are created every time the Spark program runs. So the number of files and directories keeps growing.

How can let them be deleted?

Spark version is 1.3.1 with Hadoop 2.6.

1 Answer

0 votes
by (32.3k points)

Using the "local" mode for testing purposes, you can solve this issue by creating a custom temp folder before running a test and then delete it manually (in my case I used local mode in JUnit in order to delete the temp folder automatically).

You should change the path to the temp folder for Spark by spark.local.dir property.

SparkConf conf = new SparkConf().setMaster("local")

                                .setAppName("test")

                                .set("spark.local.dir", "/tmp/spark-temp");

After the test is completed delete the /tmp/spark-temp folder manually.

Welcome to Intellipaat Community. Get your technical queries answered by top developers!

28.4k questions

29.7k answers

500 comments

94.1k users

Browse Categories

...