0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

When running "sbt package" from the command line for a small Spark Scala application, I'm getting the "value $ is not a member of StringContext" compilation error on the following line of code:

val joined = ordered.join(empLogins, $"login" === $"username", "inner")
  .orderBy($"count".desc)
  .select("login", "count")


Intellij 13.1 is giving me the same error message. The same .scala source code gets compiled without any issue in Eclipse 4.4.2. And also it works well with maven in a separate maven project from the command line.

It looks like sbt doesn't recognize the $ sign because I'm missing some plugin in my project/plugins.sbt file or some setting in my build.sbt file.

1 Answer

0 votes
by (32.5k points)

$-notation for columns can be used by importing implicit on SparkSession object (spark)

Do this:

val spark = org.apache.spark.sql.SparkSession.builder

        .master("local")

        .appName("App name")

        .getOrCreate;

import spark.implicits._

val joined = ordered.join(empLogins, $"login" === $"username", "inner")

  .orderBy($"count".desc)

  .select("login", "count")

Welcome to Intellipaat Community. Get your technical queries answered by top developers !

Categories

...