Difference between na().drop() and filter(col.isNotNull) (Apache Spark)

Question

Is there any difference in semantics between df.na().drop() and df.filter(df.col("onlyColumnInOneColumnDataFrame").isNotNull() && 

!df.col("onlyColumnInOneColumnDataFrame").isNaN()) where df is Apache Spark Dataframe?

Answer 1

With df.na.drop() you actually drop the rows containing any null or NaN values.

And With df.filter(df.col("onlyColumnInOneColumnDataFrame").isNotNull()) you drop those rows which have null only in the column onlyColumnInOneColumnDataFrame.

 

In order to achieve the same thing with df.na.drop() , you can do:

 df.na.drop(["onlyColumnInOneColumnDataFrame"])