How to load java properties file and use in Spark?

Question

I want to store the Spark arguments such as input file, output file into a Java property files and pass that file into Spark Driver. I'm using spark-submit for submitting the job but couldn't find a parameter to pass the properties file. Have you got any suggestions?

Answer 1

I found a solution to resolve your problem:

props file : (mypropsfile.conf) // note: prefix your key with "spark." else props will be ignored.

spark.myapp.input /input/path

spark.myapp.output /output/path

launch

$SPARK_HOME/bin/spark-submit --properties-file  mypropsfile.conf

how to call in code :( inside code)

sc.getConf.get("spark.driver.host")  // localhost

sc.getConf.get("spark.myapp.input")       // /input/path

sc.getConf.get("spark.myapp.output")      // /output/path