using LSTMs Decoder without teacher forcing - Tensorflow

Question

I'm trying to build a sequence to sequence model in Tensorflow, I have followed several tutorials and all is good. Until I reached a point where I decided to remove the teacher forcing in my model. below is a sample of decoder network that I'm using :

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, 

                     target_sequence_length, max_summary_length, 

                     output_layer, keep_prob):

"""

Create a decoding layer for training

:param encoder_state: Encoder State

:param dec_cell: Decoder RNN Cell

:param dec_embed_input: Decoder embedded input

:param target_sequence_length: The lengths of each sequence in the target batch

:param max_summary_length: The length of the longest sequence in the batch

:param output_layer: Function to apply the output layer

:param keep_prob: Dropout keep probability

:return: BasicDecoderOutput containing training logits and sample_id

"""

training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,

                                                    sequence_length=target_sequence_length,

                                                    time_major=False)

training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, training_helper, encoder_state, output_layer)

training_decoder_output = tf.contrib.seq2seq.dynamic_decode(training_decoder,

                                                            impute_finished=True,

                                                            maximum_iterations=max_summary_length)[0]

return training_decoder_output

As per my understanding, the TrainingHelper is doing the teacher forcing. Especially that is it taking the true output as part of its arguments. I tried to use the decoder without training help but it appears to be mandatory. I tried to set the true output to 0 but apparently the output is needed by the TrainingHelper. I have also tried to google a solution but I did not find anything related.

Answer 1

There are various Helpers which all inherit from the same class. As you said TrainingHelper requires predefined true inputs which are expected to be outputted from the decoder and this true inputs are fed as next steps (instead of feeding the output of a previous step). This approach (by some research) should speed up the training of decoder.

In your case, you are looking for GreedyEmbeddingHelper. Just replace it instead of TrainingHelper

For more information regarding the same, refer to the following link:

https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/