Perhaps too general a question, but can anyone explain what would cause a Convolutional Neural Network to diverge?
I am using Tensorflow's iris_training model with some of my own data and keep getting
ERROR:tensorflow:Model diverged with loss = NaN.
tensorflow.contrib.learn.python.learn.monitors.NanLossDuringTrainingError: NaN loss during training.
Traceback originated with the line:
tf.contrib.learn.DNNClassifier(feature_columns=feature_columns, hidden_units=[300, 300, 300],
#optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.001, l1_regularization_strength=0.00001), n_classes=11, model_dir="/tmp/iris_model")
I've tried adjusting the optimizer, using a zero for learning rate, and using no optimizer. Any insights into network layers, data size, etc is appreciated.