In this lecture we are going to go through a collage notebook that emphasizes the importance of shapes
in art ends.
Remember that whenever you hear me say something like and by t by D you should be automatically thinking
about a box without me explicitly showing you a box.
If you don't have this automatic visualization reflex you'll be at a disadvantage in trying to learn
this stuff.
This lecture is all about tracking the shapes in an art n and also we are going to go through the art
and calculation manually to reinforce our understanding of how an art and works as usual.
You can look at the title of the notebook to determine what notebook we are currently looking at.
So the first thing we have here are just some comments where I list out all the important size variables
we have to pay attention to.
These things should be permanently stored in your memory.
You should never be asking what does m mean again if you are.
That will significantly slow down your learning so take notes and write things down if you have to.
So just to recap and is the number of samples in your dataset.
This has been the case since the beginning of this course.
T is the sequence length.
Remember that pi to which we assume constant size sequences D is the input feature dimensionality.
We've gone through many examples of this where you might have a D bigger than one.
M is the number of hidden units.
This is the same as we have in a regular feed forward ANZ so it's a hyper parameter which you can choose.
Finally K is the number of output nodes as a Sino K being bigger than one does not automatically imply
you are doing classification with a soft Max.
You can do multi-dimensional regression to imagine for instance you are trying to predict lat long coordinates
in that scenario.
K would be too but it would still be a regression problem.
So next we're going to make some dummy data also going to set our size variables so we're setting and
t the K and then we make X to be just a random array of size and by TBD.
So and is one.
So for this example we're only going to be working with one sample t is 10.
So our sequence length is 10 the equals 3.
So our feature dimensionality is 3 M is equal to 5.
So our hidden feature size is 5 finally came was to so we have two APA nodes.
And as you know our input x will be a shape.
And by t by the
next we're going to create our model so as before we're going to create a custom class called simple
aren't at this time.
I'm not going to have an input argument for the number of hidden layers since the default is 1 and so
it does not need to be specified inside the constructor.
We have the same steps as before we said all the arguments to be instance variables we instantiate the
RNA module and we create a dense layer with the number of output units a K and just for fun I'll use
the 10 inch activation as the RNA in a non linearity.
Next we have the forward function which is essentially the same as before but with two crucial differences.
First we're not going to use the GP you in this notebook.
Since we're not training anything and second instead of just taking the final hit and state and passing
it through the dense layer I'm going to take all the hit and states and pass them through the final
dense layer.
So now we'll expect our output to be of size and by t by k instead of just end by K
next we're going to instantiate our model and then pass it X to get the model's output
so obviously both our data and weights are random.
So this prediction is not meaningful.
These numbers are just for sanity checking
as you can see the output shape is as expected one by ten by two.
This is because we have one sample a sequence length of 10 and two outputs taking note of these numbers
as this is what we want to compare with later on.
Next we're going to detach the output and save it as a number pi array so we can use it later.
Next we're going to grab the parameters of the RNA module
as you can see when I call the parameters function it returns for things.
Now you might ask how do I know that the parameters will be returned in this order.
Well I don't.
I had to figure it out by running this script and making sure the outputs were the same as my manual
calculation.
So as an exercise if you want to close this notebook and try to do it yourself you're strongly encouraged
to do so
if we inspect w x h by checking a shape and printing out its values.
We can verify that it really refers to the inputs ahead and wait.
If you recall the input dimensionality is three and the hidden dimensionality is five
max.
We just have some code to copy each of the parameters to an empire is.
Now if we check the shape of all the parameters our ordering seems to make sense w x h we already confirmed
w h h should be five by five which it is.
What's weird is that pi towards separates the inputs a hidden bias and the hidden to hidden bias that
won't be a problem as long as we follow the principles.
Next we grab the parameters of our final fully connected layer w o and b o if we check the shape the
ordering appears to be correct
the last step is to do our manual or an N calculation.
This just follows the pseudocode we discussed earlier.
So hopefully you were taking notes to start we're going to initialize the initial head and state to
a vector of zeros.
Next we get X at index 0 which is our one and only sample.
Next we initialize an array of zeros for our y hats.
As you know in this example we only have one sample.
So for simplicity I'm going to make it an array of size t by k.
Next we enter a loop where little T counts up from zero up to Big T inside the loop.
We first calculate H.
That's the hidden value at the hidden layer.
This calculation is a little bit different from the theory lecture since the RNA layer has two biased
terms.
Nevertheless we can follow the usual pattern in order to transform x of T.
We're going to multiply by W x H and at the bias term b x H.
Next we're going to take the previous head and state H last multiply it by W H H and add the bias term
be a change so that transforms H last.
So there are both a fine transformations on X of T and H last.
Next we add both these linear transformations together and apply the 10 H activation.
Once we have h we can calculate Y which is just the usual equation multiply by though you 0 and add
B O and we also store the Y that we get into our y hearts array.
Finally we assign h h last so that H last has the correct value for the next iteration of the loop.
Once we're outside the loop we can print the final value of the Y hat's array and hopefully this is
equal to what we calculated before with the PI torch model
now.
Since this is just a big list of numbers and most of us want memorized what we saw before it would be
better to check this programmatically by using the all closed function
awesome.
So we've confirmed that these are indeed the calculations that are done by a simple in
one thing that made this exercise simpler was that we only had one sample.
As a bonus exercise here's what you can do.
Use an N bigger than one.
Modify this code so that it still produces the same result even when you have multiple samples.