Request for example: Recurrent neural network for predicting next value in a sequence

Question

Can anyone give me a practical example of a recurrent neural network in (pybrain) python in order to predict the next value of a sequence? (I've read the pybrain documentation and there is no clear example for it I think.) I also found this question. But I fail to see how it works in a more general case. So, therefore, I'm asking if anyone here could work out a clear example of how to predict the next value of a sequence in pybrain, with a recurrent neural network.

To give an example.

Say for example we have a sequence of numbers in the range [1,7].

First run (So first example): 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6

Second run (So second example): 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6

Third run (So third example): 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7

and so on.

Now given for example the start of a new sequence: 1 3 5 7 2 4 6 7 1 3

what is/are the next value(s)

This question might seem lazy, but I think there lacks a good and decent example of how to do this with pybrain.

Additionally: How can this be done if more than 1 feature is present:

Example:

Say for example we have several sequences (each sequence having 2 features) in the range [1,7].

First run (So first example): feature1: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6

                              feature2: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7

 

Second run (So second example): feature1: 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6

                                feature2: 1 2 3 7 2 3 4 6 2 3 5 6 7 2 4 7 1 3 3 5 6    

Third run (So third example): feature1: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7

                              feature2: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6

and so on.

Now given for example the start of a new sequences:

                                            feature 1: 1 3 5 7 2 4 6 7 1 3

                                            feature 2: 1 2 3 7 2 3 4 6 2 4

what is/are the next value(s)

Feel free to use your own example as long it is similar to these examples and has some in-depth explanation.

Answer 1

Pybrain implementation required a tuple for the Unsurpervised Data Set object:

Code:

from pybrain.tools.shortcuts import buildNetwork

from pybrain.supervised.trainers import BackpropTrainer

from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet

from pybrain.structure import LinearLayer

ds = SupervisedDataSet(21, 21)

ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))

ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)

trainer = BackpropTrainer(net, ds)

trainer.trainEpochs(100)

ts = UnsupervisedDataSet(21,)

ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

Output:

[1, 2, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6, 7, 1, 4, 6, 1, 2, 2, 3, 6]

For the smaller sequences prediction, simply train it as such, that can be as sub sequences or as overlapping sequences:

from pybrain.tools.shortcuts import buildNetwork

from pybrain.supervised.trainers import BackpropTrainer

from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet

from pybrain.structure import LinearLayer

ds = SupervisedDataSet(10, 11)

z = map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split())

obsLen = 10

predLen = 11

for i in xrange(len(z)):

  if i+(obsLen-1)+predLen < len(z):

    ds.addSample([z[d] for d in range(i,i+obsLen)],[z[d] for d in range(i+1,i+1+predLen)])

net = buildNetwork(10, 20, 11, outclass=LinearLayer,bias=True, recurrent=True)

trainer = BackpropTrainer(net, ds)

trainer.trainEpochs(100)

ts = UnsupervisedDataSet(10,)

ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3'.split()))

[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

Output:

[3, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6]

Hope this answer helps.

If you wish to know supervised and unsupervised Machine Learning then visit this Machine Learning Course.