Here are a few phrases that
were generated when I gave the
neural network the sentence Lawrence went to Dublin,
and I asked it to predict the next 10 words.
We'll see the code for that in a moment,
but notice that there's a lot of repetition of words.
In this sentence, three of the last five words are wall,
and here three of the last four are ball,
and even in this one the word relations gets repeated.
This is because our LSTM was
only carrying context forward.
Let's take a look at what happens if we
change the code to be bidirectional.
By adding this line simply
defining the LSTM is bidirectional,
and then retraining,
I can see that I do converge a
bit quicker as you'll see in this chart.
After training and testing,
I now get these sentences.
They make a little bit more sense,
but there's still some repetition.
That being said, remember this is
a song where words rhyme such as ball,
all and wall, et cetera,
and as such many of them are going to show up.