Okay. We've mentioned the shape of the data
and the batches that the data is split up into.
It's important to take a look at that,
and let's dig into that next.
The inputs are three dimensional.
So for example, if we have a window size of
30 timestamps and we're batching them in sizes of four,
the shape will be 4 times 30 times 1,
and each timestamp, the memory cell input
will be a four by one matrix, like this.
The cell will also take the input of
the state matrix from the previous step.
But of course in this case,
in the first step, this will be zero.
For subsequent ones, it'll
be the output from the memory cell.
But other than the state vector,
the cell of course will output a Y value,
which we can see here.
If the memory cell is comprised of three neurons,
then the output matrix will be four by three because
the batch size coming in was
four and the number of neurons is three.
So the full output of the layer is three dimensional,
in this case, 4 by 30 by 3.
With four being the batch size,
three being the number of units,
and 30 being the number of overall steps.
In a simple RNN,
the state output H is just a copy of the output matrix Y.
So for example, H_0 is a copy of Y_0,
H_1 is a copy of Y_1, and so on.
So at each timestamp,
the memory cell gets
both the current input and also the previous output.
Now, in some cases,
you might want to input a sequence,
but you don't want to output on and you just
want to get a single vector
for each instance in the batch.
This is typically called a sequence to vector RNN.
But in reality, all you do is
ignore all of the outputs, except the last one.
When using Keras in TensorFlow,
this is the default behavior.
So if you want the recurrent layer to output a sequence,
you have to specify returns sequences
equals true when creating the layer.
You'll need to do this when you stack
one RNN layer on top of another.