Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (11.4k points)

I have a data frame with four fields. One of the field name is Status and I am trying to use a OR condition in .filter for a dataframe . I tried below queries but no luck.

df2 = df1.filter(("Status=2") || ("Status =3"))

df2 = df1.filter("Status=2" || "Status =3")

Has anyone used this before. Please help me out!

1 Answer

0 votes
by (32.3k points)

Instead of:

df2 = df1.filter("Status=2" || "Status =3")

Simply try:

df2 = df1.filter($"Status" === 2 || $"Status" === 3)

Learn Spark with this Spark Certification Course by Intellipaat.

Browse Categories