Intellipaat Back

Explore Courses Blog Tutorials Interview Questions
0 votes
2 views
in Big Data Hadoop & Spark by (11.4k points)

As a simplified example, I tried to filter a Spark DataFrame with following code:

val xdf = sqlContext.createDataFrame(Seq(
  ("A", 1), ("B", 2), ("C", 3)
)).toDF("name", "cnt")
xdf.filter($"cnt" >1 || $"name" isin ("A","B")).show()


Then it errors:

org.apache.spark.sql.AnalysisException: cannot resolve '((cnt > 1) || name)' due to data type mismatch: differing types in '((cnt > 1) || name)' (boolean and string).;


What's the right way to do it?

1 Answer

0 votes
by (32.3k points)

Your approach is just fine but you are missing brackets in your code. 

You have to parenthesize individual expressions.

Instead of:

xdf.filter($"cnt" >1 || $"name" isin ("A","B")).show()

Do:

xdf.filter(($"cnt" > 1) || ($"name" isin ("A","B"))).show()

31k questions

32.9k answers

500 comments

693 users

Browse Categories

...