0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I am using Spark 1.3.0 with python api. While transforming huge dataframes, I cache many DFs for faster execution;

df1.cache()
df2.cache()


Once use of certain dataframe is over and is no longer needed how can I drop DF from memory (or un-cache it??)?

For example, df1 is used through out the code while df2 is utilized for few transformations and after that, it is never needed. I want to forcefully drop df2 to release more memory space.

1 Answer

0 votes
by (31.4k points)

Actually, Spark automatically monitors cache usage on each node and drops out old data partitions in a least-recently-used (LRU) fashion.

But If you would like to manually remove an RDD instead of waiting for it to fall out of the cache, use the RDD.unpersist() method.

Use the following lines of code:

df1.unpersist()

df2.unpersist()

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...