Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (11.4k points)

I have a data frame with four fields. One of the field name is Status and I am trying to use a OR condition in .filter for a dataframe . I tried below queries but no luck.

df2 = df1.filter(("Status=2") || ("Status =3"))

df2 = df1.filter("Status=2" || "Status =3")


Has anyone used this before. Please help me out!

1 Answer

0 votes
by (32.3k points)

Instead of:

df2 = df1.filter("Status=2" || "Status =3")

Simply try:

df2 = df1.filter($"Status" === 2 || $"Status" === 3)

Learn Spark with this Spark Certification Course by Intellipaat.

Welcome to Intellipaat Community. Get your technical queries answered by top developers!

29.3k questions

30.6k answers

501 comments

104k users

Browse Categories

...