build.sbt: how to add spark dependencies

Question

I am trying to download spark-core, spark-streaming, twitter4j, and spark-streaming-twitter in the build.sbt file below:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.3",
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

I simply took this libraryDependencies online so I am not sure which versions, etc. to use.

Can someone please explain to me how I should fix this .sbt files. I spent a couple hours trying to figure it out but none of the suggesstion worked. I installed scala through homebrew and I am on version 2.11.8

Answer 1

I guess you are getting errors because you are mixing Scala 2.11 and 2.10 artifacts.

As you have:

scalaVersion := "2.11.8"

And then:

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

Where the 2.10 artifact is being required. Also, you are mixing Spark versions instead of using a compatible consistent version:

// spark 1.6.1

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

// spark 1.4.1

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

// spark 0.9.0-incubating

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

Now, to resolve your problem, I would suggest you folllow this build.sbt:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(

  "org.apache.spark" %% "spark-core" % sparkVersion,

  "org.apache.spark" %% "spark-streaming" % sparkVersion,

  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion

)

Also, twitter4j dependencies are added transitively by spark-streaming-twitter, therefore you don't have to add them manually.