+1 vote
1 view
in Big Data Hadoop & Spark by (1k points)

I found some code to start spark locally with:

val conf = new SparkConf().setAppName("test").setMaster("local[*]")
val ctx = new SparkContext(conf)

What does the [*] mean?

2 Answers

+1 vote
by (13.2k points)

[*]  means for all, as many threads as there are in your machine. Local  keyword is used to run spark locally. There are different parameters that can be used with it .

According to spark.apache.org 

Local[*] is used to run spark locally with  as many worker threads as logical cores on your machine.

The code you have given will specify a master url for a local to run run locally with all the threads on your machine.

0 votes
by (31.4k points)

I just want to add some additional information:

Don't run Spark Streaming programs locally with master configured as "local" or "local[ 1]". This allows only one CPU for tasks and if a receiver is working on it, there is no resource left to process the obtained data. Use at least "local[ 2]" to have more centers.

If you want to know more about Spark, then do check out this awesome video tutorial:

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...