multiple conditions for filter in spark data frames

Question

I have a data frame with four fields. One of the field name is Status and I am trying to use a OR condition in .filter for a dataframe . I tried below queries but no luck.

df2 = df1.filter(("Status=2") || ("Status =3"))

df2 = df1.filter("Status=2" || "Status =3")

Has anyone used this before. Please help me out!

Answer 1

Instead of:

df2 = df1.filter("Status=2" || "Status =3")

Simply try:

df2 = df1.filter($"Status" === 2 || $"Status" === 3)

Learn Spark with this Spark Certification Course by Intellipaat.