We are running a Spark job via spark-submit, and I can see that the job will be re-submitted in the case of failure.
How can I stop it from having attempt #2 in case of yarn container failure or whatever the exception be?
The number of retries is controlled by the following settings(i.e. the maximum number of ApplicationMaster registration attempts with YARN is considered failed and hence the entire Spark application):
spark.yarn.maxAppAttempts - Spark's own setting. Have a look on MAX_APP_ATTEMPTS:
private[spark] val MAX_APP_ATTEMPTS = ConfigBuilder("spark.yarn.maxAppAttempts") .doc("Maximum number of AM attempts before failing the app.") .intConf .createOptional
private[spark] val MAX_APP_ATTEMPTS = ConfigBuilder("spark.yarn.maxAppAttempts")
.doc("Maximum number of AM attempts before failing the app.")
One solution for your problem would be to set the yarn max attempts as a command line argument:
spark-submit --conf spark.yarn.maxAppAttempts=1 <application_name>
Certification in Digital Marketing |
Certification in Big Data Analytics |
Certification in Cloud & Devops |
Certification in Full Stack Web Development
Big Data and Data Science Master’s Course |
Data Science Architect Master’s Program |
Cloud and DevOps Architect Master’s Course |
Artificial Intelligence Engineer Master’s Course |
Microsoft Azure Certification Master’s Training
Data Science Course Online |
Big Data Hadoop Certification Training |
Machine Learning Course Online |
Artificial Intelligence Course and Training |
Salesforce Certification Training: Administrator and App Builder |
Tableau Training and Certification Course |
Python Data Science Course & Training |
Ethical Hacking Course |
Business Analyst Course |
Courses in Cyber Security
© COPYRIGHT 2011-2021 INTELLIPAAT.COM. ALL RIGHTS RESERVED.
Download Salary Trends Now !
Learn how professionals like you got up to 100% Salary Hike.