Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (11.4k points)

I want to store the Spark arguments such as input file, output file into a Java property files and pass that file into Spark Driver. I'm using spark-submit for submitting the job but couldn't find a parameter to pass the properties file. Have you got any suggestions?

1 Answer

0 votes
by (32.3k points)

I found a solution to resolve your problem:

props file : (mypropsfile.conf) // note: prefix your key with "spark." else props will be ignored.

spark.myapp.input /input/path

spark.myapp.output /output/path


$SPARK_HOME/bin/spark-submit --properties-file  mypropsfile.conf

how to call in code :( inside code)

sc.getConf.get("")  // localhost

sc.getConf.get("spark.myapp.input")       // /input/path

sc.getConf.get("spark.myapp.output")      // /output/path

Browse Categories