Error when building seq2seq model with tensorflow

Question

I'm trying to understand the seq2seq models defined in seq2seq.py in tensorflow. I use bits of code I copy from the translate.py example that comes with tensorflow. I keep getting the same error and really do not understand where it comes from.

A minimal code example to reproduce the error:

import tensorflow as tf

from tensorflow.models.rnn import rnn_cell

from tensorflow.models.rnn import seq2seq

encoder_inputs = []

decoder_inputs = []

for i in xrange(350):  

    encoder_inputs.append(tf.placeholder(tf.int32, shape=[None],

                                              name="encoder{0}".format(i)))

for i in xrange(45):

    decoder_inputs.append(tf.placeholder(tf.int32, shape=[None],

                                         name="decoder{0}".format(i)))

model = seq2seq.basic_rnn_seq2seq(encoder_inputs,

                                  decoder_inputs,rnn_cell.BasicLSTMCell(512))

The error I get when evaluating the last line (I evaluated it interactively in the python interpreter):

    >>>  Traceback (most recent call last):

      File "<stdin>", line 1, in <module>

      File "/tmp/py1053173el", line 12, in <module>

      File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/rnn/seq2seq.py", line 82, in basic_rnn_seq2seq

        _, enc_states = rnn.rnn(cell, encoder_inputs, dtype=dtype)

      File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/rnn/rnn.py", line 85, in rnn

        output_state = cell(input_, state)

      File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/rnn/rnn_cell.py", line 161, in __call__

        concat = linear.linear([inputs, h], 4 * self._num_units, True)

      File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/rnn/linear.py", line 32, in linear

        raise ValueError("Linear is expecting 2D arguments: %s" % str(shapes))

    ValueError: Linear is expecting 2D arguments: [[None], [None, 512]]

I suspect the error comes from my side :) On a sidenote. The documentation and the tutorials are really great but the example code for the sequence to sequence model (the english to french translation example) is quite dense. You also have to jump a lot between files to understand what's going on. Me at least got lost several times in the code.

A minimal example (perhaps on some toy data) of constructing and training a basic seq2seq model would really be helpful here. Somebody know if this already exist somewhere?

I have fixed the code above according @Ishamael suggestions (meaning, no errors returns) (see below), but there are still some things not clear in this fixed version. My input is a sequence of vectors of length 2 of real valued values. And my output is a sequence of binary vectors of length 22. Should my tf.placeholder code not look like the following?

tf.placeholder(tf.float32, shape=[None,2],name="encoder{0}".format(i))

tf.placeholder(tf.float32, shape=[None,22],name="encoder{0}".format(i))

I also had to change tf.int32 to tf.float32 above. Since my output is binary. Should this not be tf.int32 for the tf.placeholder of my decoder? But tensorflow complains again if I do this. I'm not sure what the reasoning is behind this.

The size of my hidden layer is 512 here.

the complete fixed code

import tensorflow as tf

from tensorflow.models.rnn import rnn_cell

from tensorflow.models.rnn import seq2seq

encoder_inputs = []

decoder_inputs = []

for i in xrange(350):  

    encoder_inputs.append(tf.placeholder(tf.float32, shape=[None,512],

                                          name="encoder{0}".format(i)))

for i in xrange(45):

    decoder_inputs.append(tf.placeholder(tf.float32, shape=[None,512],

                                         name="decoder{0}".format(i)))

model = seq2seq.basic_rnn_seq2seq(encoder_inputs,

                                  decoder_inputs,rnn_cell.BasicLSTMCell(512))

Answer 1

Most of the models (seq2seq is not an exception) expect their input to be in batches, so if the shape of your logical input is [n], then the shape of a tensor will be using it as an input to your model should be [batch_size x n]. In general, the first dimension of the shape is usually left out as None and inferred to be the batch size at runtime.

Since the logical input to seq2seq is a vector of numbers, the actual tensor shape should be [None, input_sequence_length]. So fixed code would look along the lines of:

input_sequence_length = 2; # the length of one vector in your input sequence

for i in xrange(350):  

    encoder_inputs.append(tf.placeholder(tf.int32, shape=[None, input_sequence_length],                           name="encoder{0}".format(i)))

Learn TensorFlow with the help of this comprehensive video tutorial: