How do I check for equality using Spark Dataframe without SQL Query?

Question

I want to select a column that equals to a certain value. I am doing this in scala and having a little trouble.

Heres my code

df.select(df("state")==="TX").show()

this returns the state column with boolean values instead of just TX

Ive also tried

df.select(df("state")=="TX").show() 

but this doesn't work either.

Answer 1

You can use dataframe.filter(condition)

For your code, the condition will be df(“state”) === “TX” ,

             df.filter(df("state")==="TX").show()

Answer 2

I solved the same problem using the syntax below:

df.filter(df("state")==="TX").show()

If you want to know more about Scala, then do check out this awesome video tutorial: