Is it good learning rate for Adam method?

Question

I am training my method. I got the result as below. Is it a good learning rate? If not, is it high or low? This is my result

lr_policy: "step"

gamma: 0.1

stepsize: 10000

power: 0.75

# lr for unnormalized softmax

base_lr: 0.001

# high momentum

momentum: 0.99

# no gradient accumulation

iter_size: 1

max_iter: 100000

weight_decay: 0.0005

snapshot: 4000

snapshot_prefix: "snapshot/train"

type:"Adam"

Answer 1

You should start with a higher learning rate to get out of local minima then decrease it to a quite small value to let settle down things. For this change, the step size should be 100 iterations to reduce the size of the learning rate every 100 iterations. These numbers are truly different from your problem and depend on multiple factors like your data scale.

Notice that validation loss behavior on the graph to see if you are overfitting the data. 

With respect to Machine Learning, Adam's method is considered to be a replacement optimization algorithm. This is the reason why Adam's Method is utilized for this. Thus, to study more about such techniques like Adam's method, go through the Machine Learning Tutorials.

Hope this answer helps you!