Back

Explore Courses Blog Tutorials Interview Questions
0 votes
1 view
in Big Data Hadoop & Spark by (11.4k points)

I am trying to download spark-core, spark-streaming, twitter4j, and spark-streaming-twitter in the build.sbt file below:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.3",
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"


I simply took this libraryDependencies online so I am not sure which versions, etc. to use.

Can someone please explain to me how I should fix this .sbt files. I spent a couple hours trying to figure it out but none of the suggesstion worked. I installed scala through homebrew and I am on version 2.11.8

1 Answer

0 votes
by (32.3k points)

I guess you are getting errors because you are mixing Scala 2.11 and 2.10 artifacts.

As you have:

scalaVersion := "2.11.8"

And then:

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

Where the 2.10 artifact is being required. Also, you are mixing Spark versions instead of using a compatible consistent version:

// spark 1.6.1

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

// spark 1.4.1

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

// spark 0.9.0-incubating

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

Now, to resolve your problem, I would suggest you folllow this build.sbt:

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(

  "org.apache.spark" %% "spark-core" % sparkVersion,

  "org.apache.spark" %% "spark-streaming" % sparkVersion,

  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion

)

Also, twitter4j dependencies are added transitively by spark-streaming-twitter, therefore you don't have to add them manually.

Welcome to Intellipaat Community. Get your technical queries answered by top developers!

28.4k questions

29.7k answers

500 comments

94.1k users

Browse Categories

...