‫OK.
‫In the last lecture we learn what gradient descent is.
‫In this lecture we are going to see how to use this mathematical technique to find the optimum W's and
‫B's for this.
‫We first need to understand the error function which we discussed in the last lecture.
‫Here are the five steps that we use to implement gradient descent.
‫The first step is to give random ideas to all WS and B's in the system.
‫Then we take one training example and put its x values as input to our system.
‫We process through the entire network to get one predicted value.
‫Now on this third step I told you that we measure the distance between predicted and the actual value
‫using an error function.
‫Let's see what this means.
‫Suppose we predicted an output of zero point three.
‫Whereas the actual value is zero.
‫One way of calculating error of prediction could be just to subtract these two.
‫That is finding out actual minus predicted which will be zero minus zero point three giving us minus
‫zero point three to remove this negative sign in the and focus only on the magnitude of this error.
‫We can simply put an absolute function or a squared function on top of it.
‫Basically meaning minus zero point three would become point three.
‫Or it would be squared and it will become zero point zero nine
‫these two are good measures of error.
‫But they do not work well when we are doing classification with neural networks.
‫For this purpose we use a different function
‫this function is called Cross entropy loss function.
‫It is represented by this formula.
‫He is equal to minus of Y into log y dash minus 1 minus Y log of 1 minus Y life here.
‫Y represent actual value and Y dice represents the predicted output value.
‫I know this looks complex much complex tended to edit functions that we saw in the last slide.
‫But the reason for using this is that this function does not have local minimums that is the graph of
‫this function.
‫Looks like this one on the left and not like this one on the right.
‫If a function has local minimums are gradient descent won't work properly and it might stop here.
‫Instead of finding the global minima which is hit
‫if you don't understand the last comment.
‫Don't worry about it.
‫The simple takeaway is for classification problems.
‫The added function to be used is this cross entropy at a function.
‫We can take a look at this edit function to build some intuition around it.
‫As you know in classification problems the output value is either 0 or 1.
‫So if the output value is 1 the second part of this function that is 1 minus Y.
‫This entire part will become 0 because 1 minus 1 would be 0.
‫If the actual output is 0 then the first item of this equation will become zero and only the second
‫time will remain.
‫So let's see if the actual output is 1 for this edit function to be minimum the function should be as
‫close to zero as possible.
‫Let's see if y is equal to 1.
‫Edit is minus of 1 and 2 log y that plus 1 minus 1 log 1 minus Y that the second term becomes zero.
‫So we are left with only minus of log y dash so for this error to be small minus log y that should be
‫small.
‫This implies that log y life should be large this well implies that why that should be large since our
‫predicted output is between 0 and 1.
‫Why does being large simply means that y dash should be as close to 1 as possible.
‫Similarly if actual value of output is 0.
‫The first term of this equation will be 0 so the added function remaining would be minus log of 1 minus
‫Y dash for this error to be small log of 1 minus Y Dash has to be logged implying that 1 minus Y.
‫This has to be large implying that Y should be as small as possible although I have not given you the
‫mathematical justification for using this function but I guess with these particular examples you are
‫getting the feel of how minimizing the error or loss function is trying to match the predicted output
‫value to the actual value.
‫So now you may have guessed the job of gradient descent is to find the minimum of this error function
‫that is we will make small changes to the values of weights and biases in that direction where we get
‫maximum decrease in.
‫Edit We will continue changing WS and B's they'll know for their degrees in it it is possible this is
‫all the process looks graphically
‫for ease of understanding.
‫I have represented all of DVDs on one actors and all biases on another axis and on the vertical axis
‫we have the corresponding value of error these values of error are calculated using the error function
‫okay.
‫So now let's revisit our steps to implement gradient descent.
‫Again the first step is setting random initial values of W and B.
‫Then we go forward to get predicted output value.
‫Then we put this predicted output value in our lost function to get the error prediction.
‫Now we have the error W's and B's say we have a W value between 1 and 2 a biased value between 0 and
‫minus 1 and added value near 1.
‫So we are nearly here on this graph.
‫Now in the fourth step we do backward propagation to finding direction of movement on this graph.
‫Which means we find Delta W and Delta B.
‫That is the change in WS and B's that will take us to the minimum point if you look at this graph.
‫You can probably see that by decreasing the weight and increasingly biased values we are will be moving
‫closer to the lowest point.
‫So basically we have initial WS and B's.
‫We will be updating our W W minus all four times delta W and will be updating or b to b minus alpha
‫times delta B head Alpha is called the learning rate.
‫Basically Delta W and Delta B are unit steps that we calculate using calculus.
‫Alpha is controlling the number of those tapes we take in that direction.
‫You can imagine the impact of large vs. small values of alpha if alpha is large we are taking multiple
‫steps in the direction of gradient descent.
‫This means that we can reach t bottom faster but problem with large alpha is that we can overshoot from
‫the minimum.
‫Imagine you're very near to the bottom but on the next time you take 50 steps instead of just one in
‫such a situation you will climb the curve on the other side.
‫So a large learning rate can help in faster descent but can face issue in the final stages of convergence.
‫Therefore a moderate value of learning rate is to be used.
‫You will see what value of learning rate is to be used in practical section of the schools.
‫Very well.
‫So the step to be taken in the direction of the descent is alpha times.
‫Reader W and alpha times there to be now.
‫How do we find the letter W ending the B here.
‫Delta W is the change in wait and Delta B is the change in bias.
‫Basically we will change the initially said W's and B's in the effort to reduce error.
‫Now let us see how to find Delta W and Delta B.
‫These values are formed by doing backward propagation which means we will look back in the network to
‫find out the instantaneous slope with respect to eat W and b
‫let me take an example with a single neuron to show you how this happens.
‫Otherwise the mathematics and calculus in more can get quite messy and is often overwhelming for some
‫student if you are comfortable with calculus you can look at the complete back propagation theory in
‫the link shared in the description of this lecture.
‫However I think with this simple example you will get a solid intuition of how back propagation works
‫here's a single neuron with two inputs X1 and X2 it first calculate linearly.
‫That is it will calculate the value of Z which is equal to w 1 X1 plus W2 x2 plus B1.
‫It then applies a sigmoid on this value of Z
‫this sigmoid of the is the predicted output of this neuron.
‫We used this predicted output with the actual output to get the error of this particular training example
‫so let's start with step 1.
‫Step 1 is we have to randomly initialize the values of weight and bias.
‫We have to wait and one bias.
‫We randomly initialize w W1 2 with 2 W2 is equal to 3.
‫And bias.
‫Value is equal to minus 4.
‫Now the second step is forward propagation.
‫That is we will take one training example and put the input values of that training example to get a
‫predicted output.
‫We have taken this training example in which X1 value is 10 x2 value is minus 4 and the output is 1.
‫This way is the actual output and it is equal to 1.
‫So we have the w 1 value.
‫We have X1 we have W2 x2 and B1.
‫So we can calculate z.
‫We put all these values to get a Z value of 4.
‫We apply the activation function that is the sigmoid function on this value of z.
‫Kennedy predicted output of this neuron to sigmoid of Z that is sigmoid of 4 gives a predicted output
‫of zero point nine two.
‫This predicted output value is divided x value that we will use in the edit function.
‫You can see that this values already very close to the actual output which is 1.
‫But let's see how we can improve this value.
‫Now the third step is error calculation.
‫We have the error function with us.
‫We have predicted output value.
‫That is why dash it as 0 1 9 8 2.
‫And we have the actual output value for the training example as one we put these two values and this
‫error function to get a final added value of zero point zero 0 7 9
‫now country fourth step which is back propagation the next few minutes are going to be a little heavy
‫on mathematics.
‫We will cover some basics of calculus here.
‫If you're not comfortable with this part it is still okay.
‫This is happening in the background and your software is handling this but if you have some understanding
‫of calculus looking at this example will tell you how a neuron is doing back propagation so do not worry
‫if you do not understand this because this is happening in the background and your software is handling
‫this.
‫It is good to have this infusion if you know a little bit of mathematics so let's see how to do backward
‫propagation.
‫We are at the end.
‫We have calculated it.
‫The first step is finding out the slope of error.
‫With that predicted output that is by dash this symbol here.
‫The Ebi data very nice simply means that we are finding the instantaneous slope of error with respect
‫to wide eyes keeping everything else constant.
‫So if you know calculus you can find a derivative of this function with respect to wide eyes.
‫Then is equal to 1.
‫This gives us an output of minus 1 by white ash.
‫We go further back in our network and we find out the slope of our output function with respect to Z
‫the output function is a sigmoid function.
‫The slope of sigmoid function with respect to Z is this value it is to but minus the upon value plus
‫it is about minus C the whole squared if you know differentiation you can differentiate this function
‫with respect to the and you will get this value of slope.
‫Lastly we find a differential of Z with respect to W1 W2 and B so Z was equal to w 1 times X1 plus W2
‫times x2 plus B1.
‫So when we find out the differential respect to w when we get X1 which is equal to then at this current
‫point for W2 we get x2 which is equal to minus 4 and 4 B we get a slope of 1.
‫Next comes the process of combining all of this.
‫We moved back in our network to find all these slopes but the slope we are actually interested in is
‫how does the edit function change with respect to w 1.
‫How does it change with respect to W2 and how does it change was meant to be to find the differential
‫of E respect to w one we applied chain rule which means that if you want to find differential of E respect
‫to w 1 you can instead find differential of E respect to by Dash multiplied with differential Avinash
‫with respect to the multiplied with differential of the respect to differential of w 1 we have calculated
‫all these 3 values in our last light you can see on the top here we know the value of wildlife we know
‫the value of Z for this particular training example we can put all these values and calculate this differential
‫and it comes out to be minus zero point 1 8 6
‫we can do the similar exercise for W2 and Fort B also for their differential of E respect to W2 comes
‫out to be zero point 0 7 4 6 and the differential of even spread to be comes out to be minus zero point
‫zero 1 8 6.
‫Now these three differentials are the unit steps that we are going to take in the direction of our descent.
‫These are the data w ones the land w twos and Delta beats we are going to use these delta values to
‫update our weights and biases.
‫So that we move in the direction where the final loss would be less than the loss that we had earlier.
‫This brings us to the last step last step is we have to update W and b the new W one would be previous
‫w one minus alpha times delta w one previous w one was to Alpha we have taken as five we have taken
‫a learning rate of five year and we calculated Delta w one as minus zero point 186 this updates R W
‫and value to two point ninety similarly we calculate W2 value and it comes out to be two point six and
‫we update b value and it is now minus three point nine you can compare the previous and new W on W to
‫be values earlier w one was to notice two point ninety three earlier w two was three nowadays two point
‫six earlier B was minus four.
‫Now it is minus three point nine
‫now since we have updated our WS and B values we have to go back to our step two we have to reiterate
‫we have to do forward propagation again and we will calculate the predicted output again so this is
‫the r training example x1 is 10 x2 is minus 4 y is 1 we put these values without updated rate and bias
‫this time the z values come out to be fourteen point seven when we apply our activation function on
‫this the value we get the predicted output value that is why Dash has zero point nine nine nine if you
‫remember last time we got a predicted value of zero point nine it too so clearly this is an improvement
‫over the last values of WS and B's this process is repeated several times till we get minimum error
‫if we have a lot of neurons in our network the same process is followed in forward propagation we go
‫to the end to find a predicted output value we use that predicted output value to find the loss then
‫we stepwise come back do the differentials find the individual differential values with Ed function
‫and then we update R wait and biases so that the final edit is reduced again I will repeat that I understand
‫that this lecture was a little mathematics heavy but if you have some background calculus I am sure
‫you would have understood but if you do not have any background in calculus I understand that you would
‫be facing some difficulty in following all the things that I said try to listen to this lecture again
‫if you are facing difficulty if you are still unable to follow the concept here Do not worry you can
‫still implement a neural network in a software tool all this mathematical calculation will be done by
‫this software tool and you do not have to do anything on your own.
‫That is the beauty of neural networks.
‫If you have to do it with hand it will take a lot of pain but with computers you can have millions of
‫neurons and millions of features and your computer will still be able to solve it
‫so do focus on the practical lecture that is where you will learn how to implement these neural networks
‫in this software to.