LSTM network learning

Question

I have attempted to program my LSTM (long short term memory) neural network. I would like to verify that the basic functionality is working. I have implemented a Backpropagation through time BPTT algorithm to train a single cell network.

Should a single cell LSTM network be able to learn a simple sequence, or are more than one cell necessary? The network does not seem to be able to learn a simple sequence such as 1 0 0 0 1 0 0 0 1 0 0 0 1.

I am sending the sequence 1's and 0's one by one, in order, into the network, and feeding it forward. I record each output for the sequence.

After running the whole sequence through the LSTM cell, I feed the mean error signals back into the cell, saving the weight changes internal to the cell, in a separate collection, and after running all the errors one by one through and calculating the new weights after each error, I average the new weights together to get a new weight, for each weight in the cell.

Am I doing something wrong? I would very much appreciate any advice.

Thank you so much!

Answer 1

LSTMs make small modifications to the information by multiplications and additions. With LSTMs, the information flows through a mechanism known as cell states. This way, LSTMs can selectively remember or forget things. Having only one cell (one hidden unit) is not a good idea even if you are just testing the correctness of your code. You should try 50 even for such a simple problem. You can refer to the following link for more information regarding the same:

https://arxiv.org/pdf/1503.04069.pdf

If you wish to learn about Neural Network then visit this Neural Network Tutorial.