Finding an optimum learning rule for an ANN

Question

How do you find an optimum learning rule for a given problem, say a multiple category classification?

I was thinking of using Genetic Algorithms, but I know there are issues surrounding performance. I am looking for real-world examples where you have not used the textbook learning rules, and how you found those learning rules.

Answer 1

The learning rate is the amount of change in the model during each step of the search process or the step size during the training of the model. It provides the most important hyperparameter to tune for your neural network in order to achieve a good performance of your model on your problem. Learning algorithms related to artificial neural network(ANN) and in particular for Deep Learning which involve many hyper-parameters. This link provides you practical guidance with recommendations for the most commonly used hyper-parameters in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. This paper will also tell you how you can deal with the more interesting results that can be obtained when allowing one to adjust many hyper-parameters. From that link, you can download the pdf or the postscripts of the paper.

For more information regarding the Setting the learning rate of your neural network, refer the following link: https://www.jeremyjordan.me/nn-learning-rate/