In the last couple of weeks,
you've looked at creating neural networks
to forecast time-series data.
You started with some simple analytical techniques,
which you then extend it to using
Machine Learning to do a simple regression.
From there you use the DNN that you
tweaked a bit to get an even better model.
This week, we're going to look at
RNNs for the task of prediction.
A Recurrent Neural Network,
or RNN is a neural network
that contains recurrent layers.
These are designed to
sequentially processes sequence of inputs.
RNNs are pretty flexible,
able to process all kinds of sequences.
As you saw in the previous course,
they could've been used for predicting text.
Here we'll use them to process the time series.
This example, will build an RNN that contains
two recurrent layers and a final dense layer,
which will serve as the output.
With an RNN, you can feed it in batches of sequences,
and it will output a batch of forecasts,
just like we did last week.
One difference will be that
the full input shape when
using RNNs is three-dimensional.
The first dimension will be the batch size,
the second will be the timestamps,
and the third is the dimensionality
of the inputs at each time step.
For example, if it's a univariate time series,
this value will be one,
for multivariate it'll be more.
The models you've been using to
date had two-dimensional inputs,
the batch dimension was the first,
and the second had all the input features.
But before going further,
let's dig into the RNN layers to see how they work.
What it looks like there's lots of cells,
there's actually only one,
and it's used repeatedly to compute the outputs.
In this diagram, it looks like there's lots of them,
but I'm just using the same one being
reused multiple times by the layer.
At each time step,
the memory cell takes the input value for that step.
So for example, it is zero at time zero,
and zero state input.
It then calculates the output for that step,
in this case Y0, and a state
vector H0 that's fed into the next step.
H0 is fed into the cell with X1 to produce Y1 and H1,
which is then fed into the cell at the next step
with X2 to produce Y2 and H2.
These steps will continue until
we reach the end of our input dimension,
which in this case has 30 values.
Now, this is what gives this type of architecture
the name a recurrent neural network,
because the values recur due to the output of the cell,
a one-step being fed back
into itself at the next time step.
As we saw in the NLP course,
this is really helpful in determining states.
The location of a word in
a sentence can determine it semantics.
Similarly, for numeric series,
things such as closer numbers in the series might have
a greater impact than those
further away from our target value.