0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I have a data frame with four fields. One of the field name is Status and I am trying to use a OR condition in .filter for a dataframe . I tried below queries but no luck.

df2 = df1.filter(("Status=2") || ("Status =3"))

df2 = df1.filter("Status=2" || "Status =3")


Has anyone used this before. Please help me out!

1 Answer

0 votes
by (32.5k points)

Instead of:

df2 = df1.filter("Status=2" || "Status =3")

Simply try:

df2 = df1.filter($"Status" === 2 || $"Status" === 3)

Learn Spark with this Spark Certification Course by Intellipaat.

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...