‫In this lecture, we are going to understand the concept behind how neural networks actually learn.
‫So tell this lecture, we have covered what that neural network is.
‫Now we are starting with how does a neural network work?
‫Here's a quick recap.
‫A neural network is a network of cells in our network.
‫We are going to use sigmoid neurons because these learn in a more controllable manner in a sigmoid neuron.
‫Two things happen.
‫We first multiply the input features with the weights that are represented by W and then add a bias.
‫Term b, this value
‫We name as Z or Z.
‫The second step is the application of sigmoid or the resistory function.
‫That is, the cell calculates one upon one
‫Plus e raise to power minus e
‫This value is the output of the cell.
‫And is always between zero and one.
‫This output becomes the new input for the next layer.
‫This continues till the last layer, till we get the final output as per our network.
‫Now the problem for which we are solving is this.
‫We want to find out the weights and biases of all the cells in the system so that the final output of this
‫network is as close to the actual value of the variable to be predicted.
‫For better understanding, let us just calculate the number of variables we need to calculate.
‫For this small network here, we have two neurons in the hidden layer and one neuron in the output layer.
‫And there are two input features, x1 and X2.
‫So for the first neuron we are trying to calculate W1 into X1 plus W2 into X2 plus B1 is equal
‫to Z.
‫This z will be put into the activation function.
‫That is the sigmoid function.
‫And that will be the output of this neuron.
‫Let us say that the output of this neuron is represented by a1 for the second neuron.
‫We have two new weights, W 3 and W 4.
‫We calculate this value w3 x1 plus W4 x2 plus B2.
‫This B2 is the bias of this neuron is equal to Z2.
‫We apply activation function on the Z2 to get A2. These A1 and A2
‫Are the inputs to this final output neuron for these two inputs.
‫We need two new weights, W5 and W6.
‫So the equation at this output neuron is W5 into a1 plus W6 into A2 plus B3 gives Z3.
‫When we apply activation function on this Z3, we get the predicted output from this output neuron.
‫So if you look at the variables that we need to estimate for weights, we have W1, W2, W3, W4 W5
‫and W6.
‫So we are estimating six weights. For biases
‫We have B1, B2 and B3 three Bias's.
‫So for this small network we need to establish the values of nine variables to make this neural network
‫ready for predictions.
‫Now how do we find out the values of Ws and Bs the technique followed for
‫This is called gradient descent.
‫Gradient descent is just another optimization technique to find minimum of a function.
‫There are other optimization techniques also, such as ordinarily squared, which is used in linear
‫regression.
‫But for a large number of features and complex relationships, gradient descent shows much better computational
‫performance than any other technique.
‫This means that if you have a large number of input variables and a very complex relationship between
‫input on output, gradient descent will train the model in a much faster way as compared to other optimization.
‫So let's first discuss the process followed and gradient descent in a stepwise manner.
‫We start by assigning a random weights and bias values to all the cells in our network.
‫Since all the weights and biased values are available, that is, we have randomly assigned all weight
‫and biases.
‫Our model is ready to give out output.
‫The second step is we input one training example.
‫We use the X values of the training example and calculate the final output of the network using these
‫weights and bias values.
‫Third step is that we compare the predicted values vs. the actual values and we note the difference
‫between these two using some error function.
‫E will come back to this error function later.
‫Remember that we have the actual Y value because this was a training observation.
‫So these actual values are being used to give feedback to our network that how bad it is performing.
‫The fourth step is we try to find out those weights and biases changing, which we can reduce this error.
‫Lastly, we update the values of the weights and bias.
‫And repeat this process from step two.
‫This loop goes on till no
‫Further reduction in error function can be achieved.
‫And these steps, the first step is called initialization. Here
‫We just gave some random initial values to weights and bias.
‫The second step is called forward propagation.
‫This is because in this step, we start with input values.
‫Process them in layer one.
‫Then we take the output of layer one and process it.
‫In layer two.
‫And so on.
‫Till we get one final predicted output.
‫We are simply moving forward in terms of the layers of the network.
‫So this is forward propagation.
‫The third and fourth step are called backward propagation in these steps.
‫We already have the final error function and we look backwards in our network to find out which weights
‫and biases have maximum impact on this error function.
‫Once we establish which weights and biases have maximum impact.
‫We update these weights biases slightly to reduce the error.
‫So this is the process we follow to implement gradient descent in neural networks.
‫But we are still not discussed.
‫The concept behind gradient descent
‫gradient descent is a mathematical technique which is used to find out B minimum of A function.
‫Let's see this example in the graph on the left on the x axis.
‫I have this variable on the Y axis.
‫I have a function applied on this variable.
‫This is the plot of this function.
‫Now, if you want to find out the value of X at which the function has minimum value, there are two
‫ways to it.
‫One is if you know the exact relationship between X and the function, you can use calculus to find
‫the minimum of this function.
‫But as you know, in our machine learning problems, we do not have this exact relationship.
‫So we use a second technique, which is an iterative technique.
‫In this technique.
‫We start at a random point on this plot.
‫Say we have this value of X and fx. Now instead of focusing on the whole graph, we focus only
‫on this small part of the graph and try to find out what happens if we slightly increase the value of
‫X or decrease the value of X.
‫In other words, we are trying to find out which way is the slope.
‫If the slope is negative, that is like this, we increase the value of X a little bit.
‫And then we will see that fx will also decrease.
‫Similarly, if the slope is positive, we decrease the value of X, which will slightly decrease the
‫value of fx.
‫We continue taking these small, small steps.
‫Till we reach the final minimum point when we are at this point moving either side only increases
‫the value of the function.
‫So we stop our process here.
‫This iterative technique of finding instantaneous slope, also known as gradient and slightly moving
‫down that slope that is descent is called gradient descent.
‫If you want to picturise this, you can think of yourself being on top of a hill.
‫You cannot see anything around you because it is dark and foggy.
‫Now, you want to come down the hill as fast as possible.
‫What do you do?
‫Ideally, if you could see, you would spot the closest downhill point and run to it.
‫But since you cannot see and you know, the gradient descent technique, you take a step in each direction,
‫see which direction is more down, and then you move from your current position to that new position.
‫Then you again take out your right foot, Jack, which is the direction of steep slope and move.
‫Did you keep doing this?
‫And eventually you will come downhill.
‫This is the concept behind gradient descent.
‫In the next lecture we will merge the two ideas.
‫The first is the process that neural networks use to implement gradient descent.
‫And the second idea was what gradient descent is.
‫Mathematically, we will merge these two and understand how gradient descent is helping us achieve the
‫minima in neural networks.