0 votes
1 view
in Big Data Hadoop & Spark by (11.5k points)

I want to store the Spark arguments such as input file, output file into a Java property files and pass that file into Spark Driver. I'm using spark-submit for submitting the job but couldn't find a parameter to pass the properties file. Have you got any suggestions?

1 Answer

0 votes
by (31.4k points)

I found a solution to resolve your problem:

props file : (mypropsfile.conf) // note: prefix your key with "spark." else props will be ignored.

spark.myapp.input /input/path

spark.myapp.output /output/path

launch

$SPARK_HOME/bin/spark-submit --properties-file  mypropsfile.conf

how to call in code :( inside code)

sc.getConf.get("spark.driver.host")  // localhost

sc.getConf.get("spark.myapp.input")       // /input/path

sc.getConf.get("spark.myapp.output")      // /output/path

Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...