I have constructed a CLDNN (Convolutional, LSTM, Deep Neural Network) structure for the raw signal classification task.

Each training epoch runs for about 90 seconds and the hyperparameters seem to be very difficult to optimize.

I have been researching various ways to optimize the hyperparameters (e.g. random or grid search) and found out about Bayesian Optimization.

Although I am still not fully understanding the optimization algorithm, I feel like it will help me greatly.

I would like to ask a few questions regarding the optimization task.

How do I set up the Bayesian Optimization with regards to a deep network? (What is the cost function we are trying to optimize?)

What is the function I am trying to optimize? Is it the cost of the validation set after N epochs?

Is spearmint a good starting point for this task? Any other suggestions for this task?

I would greatly appreciate any insights into this problem.