The "==" is using the equals methods which checks if the two references point to the same object. The definition of "===" depends on the context/object. For Spark , "===" is using the equalTo method
Let's pick a fragment of Spark-Scala code:
dataFrame.filter($"age" === 21)
There are a few things going on here:
The $"age" creates a Spark Column object referencing the column named age within in a dataframe. The $ operator is defined in an implicit class StringToColumn. Implicit classes are a similar concept to C# extension methods or mixins in other dynamic languages. The $ operator is like a method added on to the StringContext class.
The dataFrame.filter method takes an argument of Column, which defines the comparison to apply to the rows in the DataFrame. Only rows that match the condition will be included in the resulting DataFrame.
Referencing Spark: An important difference between “==” and “===” of Spark can be determined by the return value. For Column:
== returns a boolean
=== returns a column (which contains the result of the comparisons of the elements of two columns)