In this lecture, we going to go through a CoLab notebook that emphasizes the importance of shapes in
Arnett's.
Remember that whenever you hear me say something like and by TBD, you should be automatically thinking
about a box without me explicitly showing you a box.
If you don't have this automatic visualization reflex, you'll be at a disadvantage in trying to learn
this stuff.
This lecture is all about tracking the shapes in and in.
And also we are going to go through the brain and calculation manually to reinforce our understanding
of how an art and works.
As usual, you can look at the title of the notebook to determine what notebook we are currently looking
at.
So the first thing we have here are just some comments where I list out all the important size variables
we have to pay attention to, these things should be permanently stored in your memory.
You should never be asking what does it mean again?
If you are, that will significantly slow down your learning.
So take notes and write things down if you have to.
So just to recap, and is the number of samples in your data set?
This has been the case since the beginning of this course.
T is the sequence.
Remember that intensive flow?
We assume constant size sequences.
D is the input feature dimensionality.
We've gone through many examples of this where you might have a D bigger than one M is the number of
hidden units.
This is the same as we have in a regular feed forward.
And so it's a hyper parameter which you can choose.
Finally, K is the number of output nodes.
As a side note, K being bigger than one does not automatically imply you are doing classification with
a softmax.
You can do multidimensional regression.
To imagine, for instance, you were trying to predict lat long coordinates in that scenario would be
two, but it would still be a regression problem.
So next, we're going to make some dummy data.
Also going to set our size variables.
So we're setting A.D., OK, and then we make X to be just a random array of size and by Tibaldi.
OK, so and has one, so for this example, we're only going to be working with one sample T is ten,
so our sequence length is ten, the equals three.
So our feature dimensionality is three.
Finally, Cagle's two.
So we have two output nodes.
And as you know, our input X will be of shape and by T by the.
Next, we are going to create our model.
Here we also set em the number of hidden units to be five.
As usual, we start with an input layer whose shape is TBD.
Then we create a simple orange layer, which has the number of hidden units M.
Let's assume the default activation, which is a Tanach, finally we create a dense layer with the number
of output units K for this, I'll assume we're doing regression so there is no activation function.
And next, we are going to use our model to make a prediction.
Now, obviously, both our data and weights are random, so this prediction is not meaningful.
These numbers are really just for sanity checking.
And as you can see, the upward shape is, as expected, one by two.
So we have one sample and two output notes taken out of these numbers as this is what we want to compare
with later on.
Next, we can do a moral summary so that we can see all the layers of our own.
As expected, we have three layers, the input layer, the simple orange layer and the dense layer.
Now, we don't exactly know what parameters are stored in a simple Arnim, although we do know the mathematical
equation to get the output.
So let's just check the weights and see what they are.
So we have some idea of what's stored in their.
OK, so it looks like three big arrays.
That's actually more helpful to Prince out the shape of these array's.
And from there, we can deduce which array corresponds to which way in the simple arnet.
So as you can see, we get a three by five, a five by five and a five length vector, if you recall,
D equals three and M equals five.
So that makes sense.
The first weight is D by M, which means it's the input to hidden weight.
The second weight is MBM, which means it's the head into head and way, and the third weight is a vector
of length M, which means it's the bias term.
Now we can assign our weight variables with confidence.
So for the Lloret index one, we assign these weights W, X, W, H and B, H, notice I'm using shorthand
here so I don't use w h and since it's not that useful for the liora index to this corresponds to the
output layer.
So we assign these to the weights W, O and BAEO.
The last step is to do our manual and calculation, this just follows the pseudocode we discussed earlier.
So hopefully you were taking notes to start.
We are going to initialize the initial head and state to a vector of zeros.
By the way, if we get this wrong, then the output will be different.
So here's another way we can confirm that the initial state really is zero.
Next, we get X at index zero, which is our one and only sample.
So we call this little X. Next, we initialize an empty list for all of our Y hats.
As you know, in this example, we only care about the final we have.
But we are going to calculate them all for completeness.
Next way into a loop where little T counts up from zero up to Big T inside the loop, we calculate the
first H, that's the hidden value at the hidden layer.
It's equal to the Tanach of exacty dotted with W, X plus H last dotted with W, H, plus B.H. So you
should recognize this formula from the slides we just discussed.
And of course I know you took notes, so you should be cross referencing with those.
Once we have H we can calculate Y hat, which is just the usual neuron equation.
Finally, we assign each to last so that each last has the correct value for the next iteration of the
loop, and once we're outside the loop, we can print out the final value of the White House list.
And hopefully this is equal to what we calculated before when we called model that predicts.
All right, so you probably forgot these numbers by now.
Let's go back up and check.
So it's minus zero point seven and zero point four or five.
All right.
Minus four point seven zero point four or five, so that's pretty awesome.
We've confirmed that these are indeed the calculations that are done in the simple arnet.
OK, so one thing that made this exercise simpler was that we only had one sample, so as a bonus exercise,
here's what you can do using any bigger than one, modify this code so that it still produces the same
result even when you have multiple samples.