You can use **a simple generator** that would be implemented on top of your initial idea, it's an **LSTM network** wired to the pre-trained **word2vec** embeddings, that should be trained to predict the next word in a sentence.

**Gensim Word2Vec**

Your code syntax is fine, but you should change the number of iterations to train the model well.

The default iter = 5 seems really low to train a machine learning model. Even at least 100 iterations are just better than 5.

**For example: **

word_model = gensim.models.Word2Vec(sentences, size=100, min_count=1,

window=5, iter=100)

pretrained_weights = word_model.wv.syn0

vocab_size, emdedding_size = pretrained_weights.shape

print('Result embedding shape:', pretrained_weights.shape)

print('Checking similar words:')

for word in ['model', 'network', 'train', 'learn']:

most_similar = ', '.join('%s (%.2f)' % (similar, dist)

for similar, dist in word_model.most_similar(word)[:8])

print(' %s -> %s' % (word, most_similar))

def word2idx(word):

return word_model.wv.vocab[word].index

def idx2word(idx):

return word_model.wv.index2word[idx]

The resultant embedding matrix is saved into a pretrained_weights array which has a shape (vocab_size, emdedding_size).

**Keras model**

The loss function in your code seems invalid. When the model predicts the next word, then its a classification task.

However, the loss should be **categorical_crossentropy** or **sparse_categorical_crossentropy**. This method avoids one-hot encoding, which is pretty expensive for a big vocabulary.

**For example:**

model = Sequential()

model.add(Embedding(input_dim=vocab_size, output_dim=emdedding_size,

weights=[pretrained_weights]))

model.add(LSTM(units=emdedding_size))

model.add(Dense(units=vocab_size))

model.add(Activation('softmax'))

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

**Data preparation**

If you use sparse_categorical_crossentropy loss, then both the sentences and labels must be word indices. Short sentences must be padded with zeros to the common length.

train_x = np.zeros([len(sentences), max_sentence_len], dtype=np.int32)

train_y = np.zeros([len(sentences)], dtype=np.int32)

for i, sentence in enumerate(sentences):

for t, word in enumerate(sentence[:-1]):

train_x[i, t] = word2idx(word)

train_y[i] = word2idx(sentence[-1])

**Sample generation**

The trained model outputs the vector of probabilities, from which the next word is sampled and added to the input. Here the generated text would be better and more diverse if the next word is sampled, rather than a pick as argmax.

Here is an example of temperature based random sampling:

def sample(preds, temperature=1.0):

if temperature <= 0:

return np.argmax(preds)

preds = np.asarray(preds).astype('float64')

preds = np.log(preds) / temperature

exp_preds = np.exp(preds)

preds = exp_preds / np.sum(exp_preds)

probas = np.random.multinomial(1, preds, 1)

return np.argmax(probas)

def generate_next(text, num_generated=10):

word_idxs = [word2idx(word) for word in text.lower().split()]

for i in range(num_generated):

prediction = model.predict(x=np.array(word_idxs))

idx = sample(prediction[-1], temperature=0.7)

word_idxs.append(idx)

return ' '.join(idx2word(idx) for idx in word_idxs)

**Output:**

deep convolutional... -> deep convolutional arithmetic initialization step unbiased effectiveness

simple and effective... -> simple and effective family of variables preventing compute automatically

a nonconvex... -> a nonconvex technique compared layer converges so independent onehidden markov

a... -> a function parameterization necessary both both intuitions with technique valpola utilizes

Hope this answer helps.