Spark - “sbt package” - “value $ is not a member of StringContext” - Missing Scala plugin?

Question

When running "sbt package" from the command line for a small Spark Scala application, I'm getting the "value $ is not a member of StringContext" compilation error on the following line of code:

val joined = ordered.join(empLogins, $"login" === $"username", "inner")
  .orderBy($"count".desc)
  .select("login", "count")

Intellij 13.1 is giving me the same error message. The same .scala source code gets compiled without any issue in Eclipse 4.4.2. And also it works well with maven in a separate maven project from the command line.

It looks like sbt doesn't recognize the $ sign because I'm missing some plugin in my project/plugins.sbt file or some setting in my build.sbt file.

Answer 1

$-notation for columns can be used by importing implicit on SparkSession object (spark)

Do this:

val spark = org.apache.spark.sql.SparkSession.builder

        .master("local")

        .appName("App name")

        .getOrCreate;

import spark.implicits._

val joined = ordered.join(empLogins, $"login" === $"username", "inner")

  .orderBy($"count".desc)

  .select("login", "count")