It is not clear for me the difference between loss function and metrics in Keras. The documentation was not helpful for me.
You can simply use a loss function to optimize your model. An optimizer is used to minimize the loss function.
An accuracy metric is used to calculate the performance of your model, for example, Confusion matrix, etc.
Hope this answer helps.