-: Hello and welcome to this tutorial.
Now we are gonna make this second function
to initialize the weights,
and this one will be used to get an optimal learning
of actually these weights.
So the second function,
we're gonna call it weights_init,
and it will take as argument the object M,
which will represent the neural network.
So that's all.
And then colon,
and now let's get inside the function
to define what we want it to do.
So basically what we want it to do is initialize the weights
of the neural network
in such a way that we get an optimal learning.
So this will not seem particularly intuitive.
This is based on research papers and experiments.
We are going to initialize the weights
in a specific way that we haven't seen before,
but believe me, that will optimize the learning process.
So we will just implement it
without getting into the details of why
we initialize the weights this way.
And so we're gonna start by using a trick
which will be used later to make the distinction
between the convolution and the full connection.
Because remember, our AI will have eyes
and therefore it will have some convolutional layers,
and of course it will also have some fully connected layers,
and we will have a different initialization
of the weight for these two types of connections.
So we're gonna use this trick to separate
these two kinds of connections,
and then we'll use some if conditions
to get a different initialization
for each of these connections.
So this trick is to create a new variable
that we're gonna call class name,
and that will be equal to M, our object.
So M represents the neural network,
but it's an object.
We will see that later.
And we're gonna get a special attribute from this object,
which will be class name with double underscore first class
double underscore dot double underscore again name,
and almost there, another double underscore.
So that's a pretty ugly trick to look for the type
of connection of our neural network object,
but that will give us exactly what we want.
You're gonna see it's gonna make sense
when we start the if conditions.
And by the way, speaking of if conditions,
we can start them right now.
And so what we're gonna do now is
start a first if condition,
which will get us the first case,
that is, if the connection is a convolution.
And so to write this condition,
we write if class name dot find.
Here we use a method, the find method,
find and inside of input in quotes,
conv for convolution.
And so if class name dot find conv is,
we're gonna do different than minus one.
That is actually if we have a convolution,
because minus one means no.
Well, in that case we will do a special initialization
of the weights.
So this condition here means
if we have a convolution connection.
So in that case,
what we do is run this specific initialization
of the weights we wanna do.
And so that's where all the non-intuitive things will come.
We will start by creating a variable
that we're gonna call weight underscore shape.
So weight underscore shape will be a list
that will basically contain the shape of the weight
in our neural network M.
And so we're gonna use the list function to create a list,
and inside we're going to input M, the neural network,
dot weight, which will be the weight of the neural network,
but in the convolution connection.
And to get the shape of these weights,
we use another attribute, which is dot data
and then size.
Size will get us the shape of these weights
in the convolution connection.
So now weight shape contains
in a list the shape of the weights
and the convolution connections of our neural network M.
All right, so we have weight shape.
Then to initialize the weights
of this convolution connection,
we're gonna need two values.
First is the product of the first dimension
by the second dimension by the third dimension.
So that's what we're gonna get right now.
And then we will also need to get the zeroth dimension
times the second dimension times the third dimension,
and then we'll use these two values
in the computation of how we initialize the weights.
So let's get these two.
This first product, we call it fun in,
and that will be equal to the product,
and we're gonna use the prod function,
which is a function by NumPy,
which has the shortcut NP.
So NP, that's prod,
and inside prod we input what we want to make
the product of.
And so as we said, that is the dimension one, two,
and three of our weight shape.
And so to get this,
we can take our weight shape
and get the indexes of these three dimensions.
And so we said it's dimension one
up to dimension three included.
So up to dimension four excluded.
And that's how we can get it.
Four, the upper bound here, is not included.
So that's exactly what we want.
Then same for fun out.
As we said, fun out is going to be the product
of the dimension zero times the dimension two
times the dimension three.
And so here we can get the index from two included
to four excluded.
So that will get us the product of dimension two and three.
And then we can multiply it by the dimension zero,
which we can access with weight shape zero of index zero.
So to sum up, this is dim one times dim two times dim three,
and just below we have dim zero times dim two
times dim three of our weight shape list of weights.
All right, so now we're gonna use these two values
fun in and fun out to proceed to the initialization,
because we're going to compute a new value
that we're gonna call W bound,
and that will be equal to the square root,
which we can get with the function NP from NumPy dot SQRT,
exactly like before.
So the square root of six divided by fun in plus fun out.
So fun in plus fun out.
There we go.
So this W bound here represents
in some way the size of the tensor of weights.
And why did we get this?
It's because then what we are just about to do now
is we want to generate some random weights
that are inversely proportional to the size
of the tensor of weights,
because indeed what we're about to do now
is take our neural network M,
then get its weight,
so by still taking the attribute weight,
then access its data, that is, the tensor itself.
And then from this tensor of weights,
we're gonna generate some random weights that are
inversely proportional to the size of the tensor of weights.
And so in this uniform function now,
we have to input a lower bound,
which will be minus W bound,
and the upper bound, which will be plus W bound.
Okay, so that's for the weights.
And now we need to initialize the bias.
And good news for the bias.
It's gonna be much more simple.
We are going to initialize them all with zeros.
So to get these bias, we take them from our model,
of course, that is, our neural network.
And then the attribute for the bias is bias.
Then same we access to the data,
and then we're gonna use a method,
which is the fill underscore method,
which, as you might have guessed,
is used to fill the tensor of bias with zeros.
Well, with zeros, we have to specify that we want to fill it
with zeros here,
and so that's why I'm inputting here zero.
All right, so to summarize,
we generate some random weights inversely proportional
to the size of the tensor of weights
and we initialized the bias with zeros.
All right, so that was for the initialization
of the convolution connections,
and now we need to do the same for the full connection.
And so we're gonna add a new condition, elif.
Same we take this trick we used,
first class name that is this variable
that contains the different names of the connections.
So if class name dot same we use the find method,
to which we input in quotes this time a full connection,
that is, a classic linear connection
in a classic artificial neural network.
And so the name for that is linear.
And same, we're gonna use this trick to say
that we want it to be different than minus one
so that this elif class name find linear
different than minus one means if the connection is linear,
that is, if we have a classical connection.
So in that case, how are we going to initialize the weights?
Well, it's gonna be quite the same.
We are going to introduce a weight shape variable
which will not erase the first one because we will either be
in this case or that case,
so it will not be the same.
So we can totally reuse that.
Then same, we're gonna introduce a fun in variable,
which this time will not be equal
to the product of these three dimensions
but actually this time it will be equal
to simply the dimension one,
and that's because for the full connection
there is less connections than in a convolution connection.
You saw this in the intuition lectures in the ANN
and the CNN section.
There is less dimension for a full connection
than for a convolution.
So basically we just take this dimension one here.
Then same, we're gonna have a fun out variable
which we'll then use to compute W bound.
And this fun out dimension is going to be weight shape
of index zero,
that is, the dimension zero.
All right, then to compute W bound,
it's going to be the same.
It's going to be the square root of six
divided by the sum of fun in and fun out.
So there we go.
And then the good news is that
it's exactly the same as previously.
We used the uniform function for the weights
and the fill function for the bias
to get the same kind of initialization,
but this time with a different fun in and fun out,
and therefore a different W bound.
So that's the same principle.
That's the same idea.
The only thing that changes here is that we have
less dimensions for the full connection
and therefore more simple computation of this bound
of the weights here to generate these random weights.
But the good news is that now it's ready.
Not only this weights init function is ready,
but now we have our two tools,
and so we are ready to start building the brain.
So I can't wait.
This will be, of course, the most exciting part.
This was just to warm up
and get us ready for the big thing.
So we'll take care of that in the next tutorial.
Well, actually, it's gonna take
several tutorials, of course.
We'll start by making the eyes,
and then remember,
we'll add an LSTM to learn the temporal properties
of the input,
and then we'll take care of the actor and the critic,
and that's where we'll use
these two function normalized columns initializer
and weights init. So I can't wait to do that.
We are gonna make something very powerful now,
so get ready for it.
Until then, enjoy AI.