Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (11.4k points)

Let us consider the run() method of the BinaryClassification example:

  def run(params: Params) {
    val conf = new SparkConf().setAppName(s"BinaryClassification with $params")
    val sc = new SparkContext(conf)

Notice that the SparkConf did not provide any means to configure the SparkMaster.

When running this program from Intellij with the following arguments:

--algorithm LR --regType L2 --regParam 1.0 data/mllib/sample_binary_classification_data.txt

the following error occurs:

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set
in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:166)
    at org.apache.spark.examples.mllib.BinaryClassification$.run(BinaryClassification.scala:105)


I have also tried adding in the Spark Master url anyways (though the code seems NOT to support it ..)

  spark://   --algorithm LR --regType L2 --regParam 1.0


--algorithm LR --regType L2 --regParam 1.0 spark://

Both do not work with error:

Error: Unknown argument 'data/mllib/sample_binary_classification_data.txt'

1 Answer

0 votes
by (32.3k points)

One approach to resolve your errors is that you can set the Spark master from the command-line by adding the JVM parameter:


Also, if you want to get this done from code you can use .setMaster(...) when creating the SparkConf:

val conf = new SparkConf().setAppName("Simple Application")


And for Spark 2.x + do:

val spark = SparkSession.builder()



Browse Categories