Hello.
Welcome back.
In this lesson we shall begin a process of building our own deep learning library from scratch.
Earlier on we saw how to you know compute the cuts or no cut.
We saw how to create a model using logistic regression to predict whether an image has a cut or not.
Now we are going to use deep neural network to compute the same.
That same problem.
We're going to use deep neural network for for solving the same problem.
So in the first four or five lessons we agreed to just create a number of functions for our deep learning
library and this first network is going to be a two lay out network.
It's gonna have an input layer a single hidden layer right.
And then later on we shall create we shall other function.
I can take any size that you can just say you want five hidden layers and you'll be able to call it
with like one line of code.
So later on we shall implement.
Implement a function that accept us.
I comment in a number of layers but for this first initial one we'll take a look at two layers.
After this you get a grasp of what is required to write one that takes in a number and then create um
the layers.
So I'm gonna start off of open my idea over here.
When I press Contro s to save
and I'm going to call this to n n underscore lib V1.
Right.
So let's start off.
Okay.
So let's start off by importing num pi gonna import num pi S and P
next.
Let's start by writing a function that would initialize parameters in our parameters are the weights
and the bias W and P.
So this function is going to initialize parameters for a truly a network.
Right.
So um to take us argument the function is going to take the um the size of the input layer meaning the
number of input notes and then we can parse the um the size of the hidden layer meaning number of nodes
for the hidden layer and then number of nodes for the output layer as well.
So I'm gonna come here and say def
my tag and work working.
Sorry bother and then we're gonna call this initialize parameters
and a first argument is the size of input layer.
I call this an x.
The second argument is size of Italy layer N H four hit in the last size of our triple layer and y like
this.
So we need to initialize the weight in the base.
Okay so I'm going to I'm gonna seed I'm gonna see NPD or to random dot seed to make my resort repeatable.
And then once that is done I'm gonna say W equals w 1 Cause we have to be one in W2 I'll say w 1 Of
course
none pi dot random dot round n and then the size I want is N H and x then I'll multiply by zero point
one two points you were one here then W2 echoes the same thing actually
and P random round n this time the sizes
and Y and then N H and then multiply by two points to a 1 right so once we've done that we can create
one for the bias we have B one equals we initialize the bias with zeros and initialize that weight with
random numbers so B one of course n Pitot zeros and the size here is hidden layer by 1 and thus we've
got a surround and other brackets here next B to be true of course and pedo those
and the size.
This also a column vector this one sizes size of output length by one right.
Okay.
Um yeah.
So this all there is actually we have to make sure the function returns this so we can store we can
store this in a dictionary.
Oh see parameters create a dictionary parameters and then I'll say w one key value.
Um key value pair the p one here would hold our w 1
W-2 would hold our W-2
p. 1
she'd hold our b b 1
and then B to who would I be to like this.
Then I'll simply return dictionary return parameters does the name of the dictionary.
Right.
Okay.
So this function when we call this function and pass the um the size of the inputs the input the size
of the inputs the hidden layer and the output layer we get the weight matrices initialize for us as
well as the bias vectors also initialized.
Okay.
Once that is done we can implement the function for forward propagation.
So we're going to implement this function in two stages.
We would implement the function that calculates for Z and then another function that takes the z value
and computes the um the activation.
It takes the z value and passes it through the activation function to give us the a value right.
So I come over here def and then I'll call this function.
Linear forward
and then it's going to take the um it's gonna take activation from the previous layer or the inputs.
LEE I remember the input layer is often denoted as a zero.
So we pass this activation from the previous layer or the input layer as the input and then the abuse
the weight piece the bias and you know how to compute C you simply see shared by the.
We simply say Z equals w dot
A plus B like this.
And then what we're going to do is we going to store the A W and B values past here.
We're going to store it we're going to store it in that list known as cash.
So I'm gonna create a list here or an array if you prefer.
I'm going to store these values a WP we will use this later for computing back propagation and once
we've done this this function is going to return
it's gonna return to C as well as our cash like this.
Okay.
So once we've done this this will just compute C for us.
Now we write in a function that takes C and passes it through the activation function.
But before we do the less writes functions for our activation
we going to write one for sigmoid as one as well as one for the ratty Linea unit.
Say you have quite a number of images here.
Okay so this is the the real issue.
If we take a number and we pass it through the red blue we just compute the marks comparing the number
to 0 0.
If the number is below zero then zero be returned.
If if if it's above zero then the number will be returned.
Right.
So I'll come over here and creates red blue def
red Lou and then it takes an argument C and this is a simple us.
This is interesting.
Sorry about that.
This are simple as a U course we can use num pi dot maximum and then we just find a maximum of 0.
And the argument passed and then we can store dizzy value in cash for usage and then return a aswell
as cash like this.
Next we write a function to compute the sigmoid we already saw that in the previous example.
So simply going to type it out we see a sigmoid it takes an argument C and we compute the activation
by saying one divided by one plus non pi dot e XP minus C and this should return the um the sigmoid
and we can start to see value in a variable called cash return both the C and D a return a and then
return cash like this.
Okay so now that we've got what activation functions we can take the Zi and pass it through one of the
activation functions.
So we're going to write the activation part the function that performs the activation in such a way
that um you can decide or the user can decide whether to use red blue activation or sigmoid activation
right.
So we are going to write a function here called this linear activation forward.
C. diff
linear activation forward and issue will take three argument.
The first argument is the previous a value which in our case is the Z right.
But if it's the first layer this is going to be the C previous E and then we pass the weight as arguments
as well
and then we pass B and then activation.
So this third argument or this fourth argument is going to allow us to choose whether we want to use
a relative or a sigmoid
so we can check if they use a selected relative or sigmoid.
So I'm gonna see if
activation is sigmoid
then I'm simply going to I'm going to call our function this function here.
Leaning forward kind of call this and then I'm going to pass the argument a previous
and then W and then B and then this is going to return Zi this is going to return Zi in cash.
I'm going to store this in two variable C and linear cash.
Okay.
So then now I can take the Z and pass it through and pass it through a while sigmoid activation function.
So I'm going to call sigmoid here and then I'll pass Zi here apostasy returned from this line under
the sigmoid he's going to return a in cash.
So this is this.
Cash and cash.
We call this linear cash.
We will call this activation cost and we'll see the return is a coma.
Activation on underscore score.
Cash
like this.
Okay.
So this is the case for sigmoid.
So let's check what if the person selected realm.
So we can use the dose if elif
activation in course rally then we do the same thing but this time would pass it through the realm.
So we do excel at this
pasted over here and then it's going to return it and activation cautious were
the name of the function is relative.
Policy like this.
Right.
So we are going to take a one to cautious activation cash and linear cash um linear cash activation
cash.
They both have it.
We're gonna take the linear cash and activation cash and store it in an array or list and then we'll
return to our list as well.
We'll call this cash because
linear cash from here and activation cash
and then we return
a
in cash.
Right.
So now we have this.
So now let's write a function to compute the cost.
I'm gonna come over here.
Def codec function compute cost.
And this is going to take Y hot and then Y
and we can get the M value which is the number of training examples simply by saying m equals the true
um the label is this the labels of the M of the.
That's the labels for the training data.
Meaning the correct answer which is why we say Y shape 1 will give us give us the number of examples
we have.
So once that is done we complete the cost using the um using the equation we saw earlier
um we use this equation over here for our cost function.
We saw this earlier.
So I'm gonna implement a python
to simply say a cost equals
1 over m multiply by brackets will happen when I use N.P. dot then pass y come up you need a log.
So I say and p dot log y hot and we're gonna take the transpose of that and then all of this minus and
p dot 1 minus Y
comma and p dot we need a log log 1 minus Y hot
transpose like this.
Okay so this should work for us and what we gonna do is we going to squeeze the output and when we use
the squeeze function it makes the shape what we expect um what happens is we would end up with something
like this.
Let's say 20.
But when we squeeze this becomes simply 20 and this can't.
This is what our function can accept a function one should be able to accept this this data.
This number will be represented something like this.
But when we squeeze all dimensions it becomes just that number.
So I'm going to call the squeeze function on cost here.
Or simply say it cost
cost equals and Peter squeeze
I'm gonna pass cost in here like this and we're going to return cost.
Right.
So once that is done we can perform a while back propagation using the same back propagation equations
we saw earlier.
I'll say def we do it in two stages like we saw earlier like we did for forward propagation we do the
linear bits and then we activate the linear.
So I see a linear backward
and we're gonna pass the z over here and then the cush
so it does take DC and cash as I comment.
And what we're gonna do is we're going to extract the content of cash.
Remember within this cash we would have the previous e as well as weight and the bias.
So I'm gonna say
I'm gonna say a proof coma W and then B equals you get a cash.
Uh we get it from the cash and then we can get the no number of training examples.
Which is represented by M.
Of course previous shape X one like this.
Right.
So once that is done we can compute d w w e course one divided by M times non pi dot DC
a previous transpose like this.
Okay.
No DP e course
one divided by M times num pi dot
times num pi sum.
Let's find so these are the equations we are using
right.
We are computing T W one on D DP one here we are ready computer w one this bit here.
This is what we have over here.
DC Delta product X transpose and then our X is the previous activation we just a underscore proof.
Now we compete in DP 1 here.
Right.
So I'll say N.P. dot sum
and then I'm gonna say I'm gonna pass DC and actions equals one and then I'm going to set keep dimensions
to true
like this one of uh m times and put out some DC axis one cube dimensions.
Okay and now we do a previous
course and p dot w transpose DC right.
So now we point to return to three parameters return to a previous
W and DP Right.
So that's the linear back propagation.
Next we're going to take what we get from the linear back propagation and pass it through activation.
So I'm going to come over here to create a new function call this function
linear activation back for it like this one we have lean it backward only this one is linear backward
the activation is gonna be here
and this will take three arguments it's gonna take D a to crush the activation type whether we want
to use real to or the sigmoid so first less extract the contents of the cache we have cash over here
and in that we we would extract the linear cache and the activation cache
right.
So once we've done that we've got check the activation type selected by the user.
So I'll see if activation equals revenue then we simply compute DC first DC equals we've got a right
to function for the realm backward.
Right.
Basically need to write a function to compute the derivative of the sigmoid and you read.
So I'm gonna come over here top here and then I'll write really backward.
And this is going to take the D.A. as well as the cash as argument
and we asked extract the cash once that is done I'm gonna convert the DC to D.
Correct object simply do this by saying DC equals P dot array and then we take D a set copy of course
true here.
Like this.
Right.
And then I'll say DC
say square bracket C is less than or equal to zero.
What I want to do is when the C is less than or equal to zero then DC should be zero.
So say DC square brackets T.
Uh Z is less than or equal to zero to space this out a bit.
I'm simply going to return DC
next.
Oh right.
The um sigmoid quote on this is going to take the same argument D.A. and the cache and we start off
by getting our C
G escort from the cache and then we compute the actual sigmoid S equals one divided by into bracket
one plus and P that exponent minus C and then once that is done we can compute a derivative.
DC equals
a watch play by s watch play by one minus s and we can return DC.
Right.
So this is really backward.
And then there's the sigmoid backward.
Okay.
So now we can continue with our linear activation backward
so if Rallo is selected we're gonna use the real loop backward to function
past the a here and then the activation cache
on the other hand.
Um yeah.
So once we get a DNA we can um copy this paste on the other hand if sigmoid is selected if sigmoid is
selected we simply use the sigmoid backward
so we get dizzy we get easy but once we get it easy we've got to pass it through the um the back propagation
function.
So I'm gonna call the function here linear backward
and this is going to take DC and then the linear cache
and
should return previous to a as well as D W and DP
Right.
It can just copy this and put it on the sigmoid
so if Ray Lewis passed this bit he'll be used sigmoid is past this PTO be used it has to be Elif here
right.
So now we can return our previous a D.W. NDP
called D.A. prove
E.W. D.B.
no less right to function to update these parameters after we perform after we perform back propagation
and we compute the arrow.
We would have to update the parameters.
So I'm going to create a function here.
I'll say def update parameters.
Was going to take three argument.
The first argument is gonna be the parameters to create gradients on the learning rate alpha
and
in order to be able to use this function for future neural networks that we would develop which more
than two layers.
I'm going to make sure they use I can use um I'm going to make sure we get the number of layers from
the parameters so oh simply C L equals the length of parameters
to plenty of parameters divided by two rounded down to the nearest number.
And then once that is done I'm gonna use a loop.
I'm gonna say for l in range of capital L
and I'm gonna see parameters
and then into parameters dictionary I'm going to take the view I'm going to append the number what I
stop you want up you two etc..
I say W L plus 1
equals parameters.
W
plus one.
I'm using this crankier SDR
plus one minus the lending rate multiplied multiply by the gradient
and integrate into.
We're looking for the top you.
So let me just complete this and then I'll explain further
right.
See what we have here.
Yeah.
Square brackets close to here.
So we have a parameter w you cross the same parameter top you.
So if it is one w one a course the B 1 minus lending rate multiplied by two top you.
That is what we compute in essentially
you're gonna see right next we going to compute b the bias.
So I'll see parameters
B and then we can append without the um the B number what s b 1 b to be 3.
So we see SDR L plus one
equals
parameters.
B plus T R O plus 1.
Minus lending rate which play by
B.
And to get DP we do create NS which is going to be a dictionary DP and then we get a type of DP What
s DP 1 or DP two or DP three etc. L plus one like this
and we can return parameters
like this.
Right.
So once this is done we've got to write a function to predict for us
this function is going to take our test set which is x and then our test Labor's y.
And then the parameters which are the weight and the biases from our trained model.
So I'm gonna come over here and say def predict and I'll pass x y then parameters
and I'm gonna get the number of examples we have by saying M cause X shape
index 1 and I'm going to get the number of layers in the neural network as well by getting the parameters
divided by two rounded to the um the nearest whole number and equals
plenty of parameters
divided by two rounded down using the top with sludge because I'm around in it down to the nearest number
and then now I can perform my forward propagation to perform a forward propagation will need a function
that puts everything together right.
Perhaps we can implement this function in a way uh in our neural network file remember this function
we call in it our library file and we just put in to get up some popular functions that will be using
the screen to be another function there's gonna be another python script that would call all the functions
in this file and in that script perhaps we can include a while predict function
actually.
Let's keep it here.
Let's write another function to perform the company to forward propagation which we shall pass to this
predictive function.
And what I mean by the complete forward propagation is is a function that would take put the linear
and deactivation bit.
Okay let's write quickly.
I'm gonna say.
Def motto Forward prop or quarter's function is and this function is going to take X which is the um
the input and then parameters meaning the weight and the bias and then I'm gonna have a list of names
to list here.
Known as cache caches and then I'm gonna set a course X and then I'm going to get a length the length
of the um the length US and the number of layers.
Now.
Neural network that we pass we can do that by getting the length of the parameters then parameters
divided by two and then we round it down.
Now once that is done I'm gonna open a for loop for I in range
1 L
from I in range 1 to L 4 here.
Then we can do a print equals a and then we can we can perform Iowa linear activation so o say net activation
forward which is the function we created initially earlier
and then I'm going to pass a proof.
Remember I just took X here put it in a and then I took it and then I made it a previous.
So I'm gonna pass this a previous and then I'm gonna get the um I'm gonna get the weight from the dictionary
so I'll see parameters
the weight W
and then I'll get the right one what I stop you want or w two etc. by the person L over here like this
and then remember l is going to be incrementing It's a loop.
This should be uh
okay so we get a wait L and then I'm going to I'm gonna get a bias.
P
plus l here as well.
Right.
And then I'm going to select the activation type.
We're going to use realm for the first activation so see activation because relevant
Okay so we've got to perform the next one.
So we take what we get.
This is going to return.
Yeah.
And then cash.
Remember our linear forward unit activation forward function returns to object where we set it to returns
the activation and the cash.
And I'm going to take the activation into cash activation in cash or is it.
Yeah.
Activation in cash.
And I'm going to read it.
Return them.
But what I'm going to do is simply take the cash and spend it too cautious.
But before we do that we've gotta implement the second layer
pent up over here cautious that append
I want to append cash to cash is right.
Once this is done.
So this is going to go through from L to L minus 1.
So I'm gonna come out of the loop and compute for L O to see linear activation forward
and then I'm going to take a I'm gonna take the E returned from here I'm gonna pass I here a and then
parameters this time I'm looking forward to playing capital L
W plus SDR to L and then the as well say parameters
B plus capital to l like this and then the activation is going to be sigmoid
right.
So this also screen to return to object.
So this is going to return activation of the final layer capital L and the cash as well.
So this can be the the.
The function is going to return to final activation the activation of the final layer in the cash.
So I'll just put that in the return statement here.
Return a uh.
And um um we've got to append this cash total caches and return caches.
So I'm gonna say cautious is not append and we want to append cash
and then now we can return
El caches like this.
Right before we move on we've got a fix.
The quotation marks for access in the um the dictionary object it should be a single quotation not double.
I think I've been mixed in double and single
actually.
I think both single and double quotation is work in Python 3.
So I'll leave it if we receive an error we can come back and fix it.
Okay.
So now we can come up to a what predict function.
Now we have this.
We have fought propagation the complete forward propagation.
Okay so we are back to what predict function.
We get the number from the number of test examples we get a number of layers.
Now I'm gonna create a list called P O C P E course in p dot zeros initialized to zeros.
You're gonna be equal to number of test examples.
1 m.
So as you can see this a real vector.
And once that is done I'm going to perform the forward propagation or call our newly created function
model forward prop.
Bring it over here.
I'm going to pass the X that I use I'll pass over here pass it as the first I comment.
The second argument is gonna be the parameters that I use are passes
right and now um this is going to return.
Of course is going to return the activation of the final layer as well as the caches.
So I'm going to store this in
um probabilities
and then cautious
right.
So activation of the final layer is gonna be stored in a list called probabilities.
We're gonna take the probabilities and convert them to 0 and 1 based on thresholds.
We say if the probability is greater than zero point five converted to one if it less converted to zero.
So at best we know what it's crossed or not.
So I'll say for I in range zero comma is a long word probabilities pasted over here.
That shape the next one.
I'll see if
if probabilities
index zero I is greater than zero point five.
Then probabilities index 0.
I of course one else
probabilities index 0.
I of course you like this.
Okay.
Now we can print the accuracy if we want.
Um actually we we are going to okay.
We're returning to probabilities but we're going to start with the um the converted ones the ones that
we convert to binary so 1 we're going to storage in peace.
I'm gonna copy the S P over here and then if probability is greater than zero point five I'm gonna store
this in P then this in P S Well right we can compute total accuracy you can print it out print
accuracy
into SDR then put out some
and then we do p of course why don't we divide this by M which is the number of test examples and we
can return p s well we can return the entire list.
Okay so we've got all the functions we need right.
Okay okay.
Looks good.
So we've got all the functions we need and we would find out what we will find out if these functions
work if there are areas when we call them in our neural network function.
So in the next lesson we should start building our first neural network with two layers.
The first time that we saw is a logistic regression model.
And um this time we did it in an actual.
And in the first one is taken at a logistic regression approach using you know neural networks but this
one is our classic to lay a neural network and we shall build this in the next lesson and we shall make
use of the functions that we've created in this script here known as the end and underscore loop.
So if you have any questions just send me a message and I'll see you in the next lesson.
So before we um we move on to um to use these functions in our one next script.
Yeah.
A number of arrows.
I actually found while just looking at a file for instance or what yeah I was supposed to return the
activation by I ended up returning.
So don't make this error just fix it.
The C should be returned it should be change to a it was C here.
I meant to return it.
No see we don't need to return c c s the argument.
So this is just a typo and there's another error somewhere.
Well he had a W.
Make sure it s cup to a W if you type small w wouldn't is different from cup to a WD arguments except
the cup to a W.
So that's all there is.
Sorry about this.
Yeah okay right.
So I'll see you in the next lesson.