‫OK.
‫In the last lecture, we learn what gradient descent is.
‫In this lecture, we are going to see how to use this mathematical technique to find the optimum W's
‫and B's.
‫For this, we first need to understand the error function, which we discussed in the last lecture.
‫Here are the five steps that we use to implement gradient descent.
‫The first step is to give random values to all W's and B's in the system.
‫Then we take one training example and put its X values as input to our system.
‫We process through the entire network to get one predicted value.
‫Now, on this third step, I told you that we measure the distance between predicted and the actual
‫value using an error function.
‫Let's see what this means.
‫Suppose we predicted an output of zero point three.
‫where is the actual value is zero.
‫One way of calculating error of prediction could be just to subtract these two.
‫That is finding out actual minus predicted, which will be zero minus zero point three, giving us minus
‫zero point three.
‫To remove this negative sign in the error and focus only on the magnitude of this error, we can simply
‫put an absolute function or a square function on top of it, basically meaning minus zero point three
‫would become point three or it will be squared.
‫And it will become zero point zero nine.
‫These two are good measures of error, but they do not work well when we are doing classification with
‫neural networks.
‫For this purpose, we use a different function.
‫This one is called Cross Entropy Lost Function.
‫It is represented by this formula.
‫e is equal to minus f y into log y dash, minus one minus 5 log of one minus y dash.
‫y represents actual value and why dash represents the predicted output value.
‫I know this looks complex, much complex then the two error functions that we saw in the last slide.
‫But the reason for using this is that this function does not have local minimas.
‫That is the graph of this function looks like this one on the left and not like this one on the right.
‫If a function has local minimas our gradient, descent won't work properly and it might stop here instead
‫of finding the global minima which is here.
‫If you don't understand the last comment, don't worry about it.
‫The simple takeaway is for classification problems.
‫The error function to be used is this cross entropy error function.
‫We can take a look at this edit function to build some intuition around it.
‫As you know, in classification problems, the output value is either zero or one.
‫So if the output value is one, the second part of this function that is one minus Y, this entire tab
‫will become zero because one minus one would be zero if the actual output is zero.
‫Then the first term of this equation will become zero.
‫And only the second time will remain.
‫So let's say if the actual output is one for this error function to be minimum.
‫The function should be as close to zero as possible.
‫Let's see if Y is equal to one error is minus of one into
‫Log y dash plus one minus one log, one minus Y dash the second time becomes zero.
‫So we are left with only minus of log y dash.
‫So for this error to be small minus log, y dash should be small.
‫This implies that log y dash should be large.
‫This further implies that y dash should be large.
‫Since our predicted output is between zero and one, y dash being large simply means that white dash
‫should be as close to one as possible
‫Similarly, if actual value of output is zero, the first term of this equation will be zero.
‫So the error function remaining would be minus logoff one minus y dash.
‫For this error to be small, logoff one minus Y dash has to be large, implying that one minus y dash
‫has to be large, implying that Y should be as small as possible.
‫Although I am not given you the mathematical justification for using this function, but I guess with
‫these particular examples, you are getting the feel of how minimizing the error or loss function is
‫trying to match the predicted output value to the actual value.
‫So now you may have guessed the job of gradient descent is to find the minimum of this error function,
‫that is, we will make small changes to the values of weights and biases in that direction where we
‫get maximum decrease in error.
‫We will continue changing W's and B's.
‫till no further decrease in error is possible.
‫This is how the process looks graphically.
‫For ease of understanding, I have represented all of the weights on one axis and all of the biases on another axis
‫and on the vertical axis.
‫We have the corresponding value of error
‫These values of error are calculated using the error function.
‫OK.
‫So now let's revisit our steps to implement gradient descent.
‫Again, the first step is setting a random initial values of W and B, then we go forward to get predicted
‫output value.
‫Then we put this predicted output value in our loss function to get the error prediction.
‫Now we have the error.
‫W's and B's say we have a W value between one and two.
‫A biased value between zero and minus one and an error value near 1
‫So we're nearly here on this graph.
‫Now, in the fourth step, we do backward propagation to find the direction of movement on this graph.
‫Which means we find Delta W and Delta B..
‫That is the change in W's and B's that will take us to the minimum point.
‫If you look at this graph, you can probably see that by decreasing the weight and increasing the bias
‫values.
‫we'll be moving closer to the lowest point.
‫So basically, we have initial W's and B's.
‫We will be updating our W to W minus Alpha Times DELTA W.
‫And we'll be updating our B to B minus Alpha Times, Delta B.
‫Head alpha is called the Learning Rate.
‫Basically, Delta W and Delta B are unit steps that we calculate using calculus.
‫Alpha is controlling the number of those steps we take in that direction.
‫You can imagine the impact of large versus small values of Alpha. If alpha is large
‫We are taking multiple steps in the direction of gradient descent.
‫This means that we can reach the bottom faster.
‫But problem with large alpha is that we can overshoot from the minimum.
‫Imagine you're very near to the bottom.
‫But on the next term you take 50 steps.
‫Instead of just one.
‫In such a situation, you will climb the curve on the other side.
‫So a large learning rate can help in faster dissent, but can face issue in the final stages of convergence.
‫Therefore, a moderate value of learning rate is to be used.
‫You will see what value of learning it is to be used in practical section of this course.
‫Very well.
‫So the steps to be taken in the direction of the descent is alpha times delta W and Alpha times delta
‫B
‫Now, how do we find delta W and delta B
‫Delta W. is the change in weight and Delta B is the change in bias.
‫Basically, we will change.
‫They initially set W's and B's in the effort to reduce error.
‫Now, let us see how to find Delta W. and Delta B. These values are formed by doing backward propagation,
‫which means we will look back in the network to find out the instantaneous slope with respect to each
‫W and B..
‫Let me take an example with a single neuron to show you how this happens.
‫Otherwise, the mathematics and calculus involved can get quite messy and is often overwhelming for some
‫student.
‫If you're comfortable with calculus, you can look at the complete back propagation theory in the link
‫shared in the description of this lecture.
‫However, I think with this simple example, you will get a solid intuition of how back propagation
‫works.
‫Here's a single neuron with two inputs, X1 and x2.
‫It first calculate linearly.
‫That is, it will calculate the value of Z.
‫Which is equal to W one, X one plus W2 x2 plus B1.
‫It then appies a sigmoid on this value of z
‫This sigmoid of z is the predicted output of this neuron.
‫We used this predicted output with the actual output to get the error of this particular training example.
‫So let's start with step one.
‫Step one is we have to randomly initialize the values of weights and bias
‫We have two weights.
‫And one bias we randomly initialize w one to be two, W2 is equal to three and bias value is equal to
‫minus four.
‫Now, the second step is forward propagation.
‫That is, we will take one training example and put the input values of that training example to get
‫a predicted output.
‫We have taken this training example in which X1 value is 10 x two, value is minus four and the output
‫is one.
‫This Y is the actual output.
‫And it is equal to one.
‫So we have the W one value.
‫We have X1.
‫We have W2, x2 and B1.
‫So we can calculate Z.
‫We put all these values to get a Z value of 4.
‫We apply the activation function.
‫That is the sigmoid function on this value of z.
‫To get predicted output of this neuron so sigmoidal z, that is sigmoid of 4 gives a predicted output
‫of 0.982
‫This predicted output value is the y dash value that we will use in the error function.
‫You can see that this value is already very close to the actual output, which is one.
‫But let's see how we can improve this value.
‫Now, the third step is error calculation.
‫We have the error function with us.
‫We have predicted output value.
‫That is y dash as zero point nine eight two.
‫And we have the actual output value for the training example.
‫As one, we put these two values.
‫in this error function to get the final error value of zero point zero zero seven nine.
‫Now, comes the fourth step, which is back propagation.
‫The next few minutes are going to be a little heavy on mathematics.
‫We will cover some basics of calculus here.
‫If you're not comfortable with this part, it is still OK.
‫This is happening in the background and your software is handling this.
‫But if you have some understanding of calculus, looking at this example, we'll tell you how a neuron
‫is doing.
‫Back propagation.
‫Do not worry if you do not understand this, because this is happening in the background and your software
‫is handling this.
‫It is good to have this intuition if you know a little bit of mathematics.
‫So let's see how to do backward propagation.
‫We are at the end.
‫We have calculated error, the first first step is finding out the slope of error with a predicted output.
‫That is y dash.
‫This symbol here.
‫Delta e by Delta y dash simply means that we are finding the instantaneous slope of error.
‫With respect to y dash, keeping everything else constant.
‫So if you know calculus, you can find a derivative of this function with respect to y dash when
‫y is equal to one.
‫This gives us an output of minus one by y dash.
‫We go further back in our network and we find out the slope of our output function with respect to
‫Z.
‫The output function is a sigmoid function.
‫The slope of sigmoid function with respect to Z is this value.
‫e raised to the power minus Z upon one plus e raise to the power minus C Zwhole square.
‫If you know differentiation, you can differentiate this function with respect to z and you will get
‫this value of slope.
‫Lastly, we find a differential of Z with respect to W1, W2 and B. So Z was equal to W one times X1
‫plus W two times X2 plus B1.
‫So when we find out the differential respect to W1 we get X1, which is equal to 10 at this current
‫point for W2, we get X2, which is equal to minus four and four B. B we get a slope of one.
‫Next comes the process of combining all of this.
‫We moved back in our network to find all these slopes.
‫But the slope we are actually interested in is how does the error function change with respect to W1
‫How does it change with respect to W2?
‫And how does it change with respect to B to find the differential of E respect to W one?
‫We apply gene pool, which means that if you want to find differential of E respect to W one, you can
‫instead find differential of E restricted by Desh multiplied with the eventual abiders, respect to
‫the multiplied with differential of ze respect to differential of W one.
‫We have calculated all these three values in our last slide.
‫You can see on the top here we know the value of white ash.
‫We know the value of Z for this particular training example.
‫We can put all these values and calculate this differential and it comes out to be minus zero point
‫one eight six.
‫We can do the similar exercise for W2 am for.
‫Also for the differential of E respect to W2 comes out to be zero point zero seven four six and the
‫differential of evils to be comes out to be minus zero point zero one eight six.
‫Now, these three differentials are the unit steps that we are going to take in the direction of our
‫descent.
‫These are the Delta W ones, the line W 2s and Delta beats.
‫We are going to use these Delta values to update our rates and biases.
‫So that we move in the direction we are defined, the loss would be less than the loss that we had earlier.
‫This brings us to the last step.
‫Last step is we have to update W and B, the new W one would be previous W one minus Alpha Times, Delta
‫W one previous W one was two Alpha.
‫We have taken as five.
‫We have taken a learning rate of five year and we calculated Delta W one as minus zero point one eight
‫six.
‫This updates are w in value to two point ninety.
‫Similarly, we calculate W2 value and it comes out to be two point six.
‫And we update B value and it is now minus three point nine.
‫You can compare the previous and new W1 W2 B values earlier W one was two.
‫Nowadays, two point ninety earlier, W two was three.
‫Nowadays, two point six.
‫Earlier B was minus four.
‫Nowadays, minus three point nine.
‫Now, since we have updated our W's and B values, we have to go back to step two.
‫We have to reiterate, we have to do forward propagation again and we will calculate the predicted output
‫again.
‫So this is the training example, X one is ten, x two is minus four, Y is one, we put these values
‫with our updated weights and byas.
‫This time, Disney values come out to be fourteen point seven when we apply our activation function
‫on this evalu we get the predicted output value.
‫That is why dash as zero point nine nine nine.
‫If you remember last time we got a predicted value of zero point nine eight two.
‫So clearly this is an improvement over the last values of abusing these.
‫This process is repeated several times.
‫They'll be get minimum error.
‫If we have a lot of neurons in our network, the same processes followed in forward propagation, we
‫go to the end to find our predicted output value.
‫We use that predicted output value to find the loss.
‫Then we step wise.
‫Come back, do these differentials, find the individual differential values with ed function, and
‫then we update our VATE and biases so that the final edit is reduced.
‫Again, I will repeat that I understand that this lecture was a little mathematics heavy, but if you
‫have some background calculus, I'm sure you would have understood it.
‫But if you do not have any background in calculus, I understand that you would be facing some difficulty
‫in following all the things that I said.
‫Try to listen to this lecture again.
‫If you are facing difficulty, if you're still unable to follow the concept here.
‫Do not worry.
‫You can still implement a neural network in a software tool.
‫All this mathematical calculation will be done by this software tool.
‫And you do not have to do anything on your own.
‫That is the beauty of neural networks.
‫If you have to do it with hand, it will take a lot of pain.
‫But with computers, you can have millions of neurons and millions of features and your computer will
‫still be able to solnik.
‫So do focus on the practical lecture.
‫That is where you will learn how to implement these neural networks in this offer to.