What is the role of TimeDistributed layer in Keras?

Question

I am trying to grasp what TimeDistributed wrapper does in Keras.

I get that TimeDistributed "applies a layer to every temporal slice of an input."

But I did some experiments and got the results that I cannot understand.

In short, in connection to the LSTM layer, TimeDistributed, and just the Dense layer bear the same results.

model = Sequential()

model.add(LSTM(5, input_shape = (10, 20), return_sequences = True))

model.add(TimeDistributed(Dense(1)))

print(model.output_shape)

model = Sequential()

model.add(LSTM(5, input_shape = (10, 20), return_sequences = True))

model.add((Dense(1)))

print(model.output_shape)

For both models, I got the output shape of (None, 10, 1).

Can anyone explain the difference between TimeDistributed and Dense layer after an RNN layer?

Answer 1

In keras, when you build a sequential model, the second dimension is related to a time dimension. This means that if your data is of 5-dim with (sample, time, width, length, channel), then you can apply a convolutional layer using TimeDistributed (which is applicable to 4-dim with (sample, width, length, channel)) along a time dimension in order to obtain the 5-d output.

In keras version 2.0, Dense is by default applied to only last dimension (e.g. if you apply Dense(10) to input with shape (n, m, o, p) you'll get output with shape (n, m, o, 10)) that’s why in your case Dense and TimeDistributed(Dense) are equivalents.