LSTM time sequence generation using PyTorch

Question

For several days now, I am trying to build a simple sine-wave sequence generation using LSTM, without any glimpse of success so far.

I started from the time sequence prediction example

All what I wanted to do differently is:

• Use different optimizers (e.g RMSprob) than LBFGS
• Try different signals (more sine-wave components)

This is the link to my code. "experiment.py" is the main file

What I do is:

• I generate artificial time-series data (sine waves)
• I cut those time-series data into small sequences
• The input to my model is a sequence of time 0...T, and the output is a sequence of time 1...T+1

What happens is:

• The training and the validation losses goes down smoothly
• The test loss is very low
• However, when I try to generate arbitrary-length sequences, starting from a seed (a random sequence from the test data), everything goes wrong. The output always flats out

I simply don't see what the problem is. I am playing with this for a week now, with no progress in sight. I would be very grateful for any help.

Thank you

Answer 1

This is normal behavior and happens because your network is too confident of the quality of the input and doesn't learn to rely on the past (on its internal state) enough, relying solely on the input. When you apply the network to its own output within the generation setting, the input to the network is not as reliable as it was in the training or validation case where it got the true input.

Study the Recurrent Neural Network for more details on this.

I have two possible solutions for you: 

• The first is the simplest but less intuitive one: Add a little bit of Gaussian noise to your input. 
• The second is the most obvious solution: during training, feed it not the true input but its generated output with a certain probability p. Start out training with p=0 and step by step increase it in order that it learns to general longer and longer sequences, severally.