In this lecture, we are going to go through a Colau notebook that emphasizes the importance of shapes
in our own ends.
Remember that whenever you hear me say something like and by TBD, you should be automatically thinking
about a box without me explicitly showing you a box.
If you don't have this automatic visualization reflex, you'll be at a disadvantage in trying to learn
this stuff.
This lecture is all about tracking the shapes in an art in.
And also, we are going to go through the art and calculation of manually to reinforce our understanding
of how an art and works.
As usual, you can look at the title of the notebook to determine what notebook we are currently looking
at.
So the first thing we have here are just some comments.
I list out all the important size variables we have to pay attention to.
These things should be permanently stored in your memory.
You should never be asking what does m mean again?
If you are, that will significantly slow down your learning.
So take notes and write things down if you have to.
So just to recap and is the number of samples in your dataset, this has been the case since the beginning
of this course.
T is the sequence like, remember that in TensorFlow, we assume constant size sequences D is the input
feature dimensionality.
We've gone through many examples of this where you might have a d bigger than one m is the number of
hidden units.
This is the same as we have in a regular feed forward, and so it's a hyper parameter which you can
choose.
Finally, K is the number of output nodes.
As a side note, K being bigger than one does not automatically imply you were doing classification
with a soft max.
You can do multi-dimensional regression to imagine, for instance, you are trying to predict lat long
coordinates.
In that scenario, K would be two, but it would still be a regression problem.
So next, we're going to make some dummy data.
Also going to set our size variables.
So we're setting A. the OK and then we make X to be just a random array of size end by TBD.
OK, so and as one, so for this example, we're only going to be working with one sample.
Is 10, so our sequence length is 10, the equals three.
So our feature dimensionality is three.
Finally, K equals two.
So we have two ALPA nodes.
And as you know, our Input X will be a shape and by T by the.
Next, we're going to create our model.
Here we also set em the number of hidden units to be five.
As usual, we start with an input layer whose shape is TBD.
Then we create a simple orange layer, which has the number of hidden units M.
Let's assume the default activation, which is a 10h finally, we create a dense layer with a number
of output units.
OK.
For this, I'll assume we're doing regression.
So there is no activation function.
Next, we are going to use our model to make a prediction.
Now, obviously, both our data and weights are random.
So this prediction is not meaningful.
These numbers are really just for sanity checking.
And as you can see, the output shape is as expected, one by two.
So we have one sample in two output nodes taking note of these numbers as this is what we want to compare
with later on.
Next, we can do a model summary so that we can see all the layers of our own in.
As expected, we have three layers the input layer, the simple orange layer and the dense layer.
Now, we don't exactly know what parameters are stored in a simple answer, although we do know the
mathematical equation to get the output.
So let's just check the weights and see what they are.
So we have some idea of what's stored in there.
OK, so it looks like three big arrays.
And it's actually more helpful to prints out the shape of these arrays.
And from there, we can deduce which array corresponds to which way in the simple answer.
So as you can see, we get a three by five, a five by five and a five length vector.
If you recall, De equals three and M equals five.
So that makes sense.
The first way is d by M, which means it's the input to hidden weight.
The second weight is m by M, which means it's the head into and way.
And the third weight is a vector of length m, which means it's the bias term.
Now we can assign our weight variables with confidence.
So for the Lloret index one, we assign these weights W X W H and B H.
Notice I'm using shorthand here, so I don't use W X H and W h, since it's not that useful for the
lire index to.
This corresponds to the output layer, so we assign these to the weights w o and B O.
The last step is to do our manual art and calculation.
This just follows the pseudocode we discussed earlier, so hopefully you were taking notes.
To start, we are going to initialize the initial hidden state to a vector of zeros.
By the way, if we get this wrong, then the output will be different.
So here's another way we can confirm that the initial state really is zero.
Next, we get X at index zero, which is our one and only sample, so we call this little x.
Next, we initialize an empty list for all of our Y hats.
As you know, in this example, we only care about the final Y have.
But we are going to calculate them all for completeness.
Next, we into a loop where little two counts up from zero up to Big T inside the loop, we calculate
the first H.
That's the hidden value at the hidden layer.
It's equal to the 10h of exact T dotted with W X plus h last dotted with w h plus b h.
So you should recognize this formula from the slides we just discussed.
And of course, I know you took notes, so you should be cross-referencing with those.
Once we have h, we can calculate y hat, which is just the usual neuron equation.
Finally, we assign each to each last so that each last has the correct value for the next iteration
of the Loop.
And once we're outside the loop, we can print out the final value of the white list and hopefully this
is equal to what we calculated before when we call the model that predicts.
All right, so you probably forgot these numbers by now.
Let's go back up and check, so it's minus 0.7 and zero point four or five.
All right.
Minus 0.7, 0.4, five.
So that's pretty awesome.
We've confirmed that these are indeed the calculations that are done in the simple art and.
OK, so one thing that made this exercise simpler was that we only had one sample, so as a bonus exercise,
here's what you can do.
Use an end bigger than one.
Modify this code so that it still produces the same result even when you have multiple samples.