Last week, we looked at doing
classification using texts and trying to
train and understand
positive and negative sentiment in movie reviews.
We finished by looking at the effect of tokenizing words,
and saw that our classifier failed
to get any meaningful results.
The main reason for this was that the context
of words was hard to follow when
the words were broken down into
sub-words and the sequence in which
the tokens for the sub-words appear
becomes very important in understanding their meaning.
Let's take a look at that now.
The neural network is like
a function that when you feed it in data and labels,
it infers the rules from these,
and then you can use those rules.
So it could be seen as a function a little bit like this,
you take the data and you take the labels,
and you get the rules.
But this doesn't take any kind of sequence into account.
To understand why sequences can be important,
consider this set of numbers.
If you've never seen them before,
they're called the Fibonacci sequence.
So let's replace the actual values
with variables such as n_0,
n_1 and n_2, etc., to denote them.
Then the sequence itself can be derived
where a number is the sum of the two numbers before it.
So 3 is 2 plus 1,
5 is 2 plus 3,
8 is 3 plus 5, etc.
Our n_x equals n_x minus 1,
plus n_x minus 2,
where x is the position in the sequence.
Visualized, it might also look like this,
one and two feed into
the first function and three comes out.
Two gets carried over to the next,
where it's fed in along with the three to give us a five.
The three is carried on to the next where it's fed into
the function along with the five to
get an eight and so on.
This is similar to the basic idea of
a recurrent neural network or RNN,
which is often drawn a little like this.
You have your x as in input and your y as an output.
But there's also an element that's fed
into the function from a previous function.
That becomes a little more clear when
you chain them together like this,
x_0 is fed into the function returning y_0.
An output from the function is
then fed into the next function,
which gets fed into the function along
with x_2 to get y_2,
producing an output and continuing the sequence.
As you can see,
there's an element of x_0
fed all the way through the network,
similar with x_1 and x_2 etc.
This forms the basis of
the recurrent neural network or RNN.
I'm not going to go into detail and how they work,
but you can learn much more about
them at this course from Andrew.