‫In this lecture we are going to understand the concept behind how neural networks actually learn.
‫Still this lecture we have called what the neural network is now we are starting with.
‫How does a neural network work.
‫Here is a quick recap.
‫A neural network is a network of cells in our network.
‫We are going to use sigmoid neurons because these learn in a more controllable manner in a sigmoid neuron.
‫Two things happen.
‫We first multiply the input features we deviate that are represented by W and then add a bias to be
‫this value.
‫We name as Z or Z.
‫The second step is the application of sigmoid or the logistic function that is the cell calculate one
‫upon one plus it is the part minus the.
‫This value is the output of the cell and is always between 0 and 1.
‫This output becomes the new input for the next layer.
‫This continues till the last layer till we get the final output.
‫As per our network now the problem for which we are solving is this we want to find out the ways and
‫biases of all the cells in the system so that the final output of this network is as close to the actual
‫value of the variable to be predicted
‫for better understanding.
‫Let us just calculate the number of variables we need to calculate for this small network head we have
‫two neurons in the head and left and one neuron in the output layer and there are two input features
‫x1 and x2.
‫So for the first neuron we are trying to calculate W1 into X1 plus W2 into x2 plus B1 is equal to Z.
‫This V will be put into the activation function that is the sigmoid function and that will be the output
‫of this neuron.
‫US say that the output of this neuron is represented by A1 for the second neuron.
‫We have two new Newgate W3 and w4.
‫We calculate this value W3 X1 plus W for x2 plus B2 this B2 is the bias of this neuron is equal to z2.
‫We apply activation function on the z2 to get it to these A1 and A2 are the inputs to this final output
‫neuron for these two inputs.
‫We need two new rate W5 and W six.
‫So the equation at this output neuron is W5 into even plus W 6 into A2 plus b 3 gives z3.
‫When we apply activation function on this Z3 we get the predicted output from this output neuron.
‫So if we look at the variables that we need to estimate for weights we have W1 W2 W3 w for w 5 and W
‫6.
‫So we are estimating 6 wait for biases we have B1 B2 and B3 3 biases.
‫So for this small network we need to establish the values of nine variables to make this neural network
‫ready for predictions.
‫Now how do we find out the values of WS and beats the technique followed for.
‫This is called gradient descent gradient descent is just another optimization technique to find minimum
‫of a function.
‫There are other optimization techniques also such as ordinarily squared which is used in linear regression
‫but for a large number of features and complex relationships gradient descent shows much better computational
‫performance than any other technique.
‫This means that if you have a large number of input variables and a very complex relationship between
‫input and output gradient descent will train the model in a much faster way as compared to other optimization
‫so let's first discuss the process followed in gradient descent in a stepwise manner.
‫We start by assigning a random weight and bias values to all this is in our network.
‫Since all the weights and biased values are available that is we have randomly assigned all the weight
‫and biases.
‫Our model is ready to give out output.
‫The second step is the input one training example.
‫We use the x values of the training example and calculate the final output of the network using these
‫weights and biased values.
‫Third step is that we compare the predicted values vs. the actual values and we know the difference
‫between these two using some error function.
‫E will come back to this error function later.
‫Remember that we have the actual y value because this was a training observation so these actual values
‫are being used to give feedback to our network that how bad it is performing
‫the fourth step is we try to find all those weights and biases changing which we can reduce this error.
‫Lastly we update the values of the weights and bias and repeat this process from step two.
‫This loop goes on the low for the reduction and edit function can be achieved and these steps.
‫The first step is called initialization head.
‫We just give some random initial values to base and bias.
‫The second step is called forward propagation.
‫This is because in this step we start with input values process them in Layer 1.
‫Then we take the output of Layer 1 and process it and layer 2 and so on.
‫Then we get one final predicted output.
‫We are simply moving forward in terms of the layers of the network.
‫So this is forward propagation.
‫The third and fourth step are quite backward propagation in these steps.
‫We already have the final edit function and we look backwards in our network to find out which weeds
‫and biases have maximum impact on this error function.
‫Once we establish which weights and biases have maximum impact.
‫We update these weights and biases slightly to reduce the edit.
‫So this is the process we follow to implement gradient descent and neural networks but we are still
‫not discussed the concept behind gradient descent
‫gradient descent is a mathematical technique which is used to find out the minimum of a function.
‫Let's see this example in the graph on the left on the x axis.
‫I have this variable and on the y axis I have a function applied on this variable.
‫This is the plot of this function.
‫Now if you want to find out the value of X at which the function has minimum value there are two ways
‫to it.
‫One is if you know the exact relationship between X and the function you can use calculus to find the
‫minimum of this function.
‫But as you know in r machine learning problems we do not have this exact relationship.
‫So we use a second technique which is an attractive technique in this technique.
‫We start at a random point on this block saying we have this value of x and ethics.
‫Now instead of focusing on the whole graph we focus only on this small part of the graph and try to
‫find out what happens if we slightly increase the value of X or decrease the value of x.
‫In other words we are trying to find out which way is this slope.
‫If this globe is negative that is like this we increase the value of X a little bit and then we will
‫see that if X will also decrease.
‫Similarly if the slope is positive we decrease the value of x which will slightly decrease the value
‫of Epix
‫we continue taking these small small steps till we reach the final minimum point when we are at this
‫point moving either side only increases the value of the function so we stop our process here.
‫This outdated technique of finding instantaneous slope also known as gradient and slightly moving down
‫that slope that is descent is called gradient descent.
‫If you want to picture this you can think of yourself being on top of a hill.
‫You cannot see anything around you because it is dark and foggy.
‫Now you want to come down the hill as fast as possible.
‫What do you do.
‫Ideally if you could see you would spot the closest down hill point and run do it.
‫But since you cannot see and you know the gradient descent technique you take a step in each direction
‫see which direction is more down and then you move from your current position to that new position.
‫Then you again Take out your right foot check which is the direction of steep slope and move.
‫Did you keep doing this and eventually you will come downhill.
‫This is the concept behind gradient descent.
‫In the next lecture we will merge the two ideas.
‫The first is the process that neural networks used to implement gradient descent.
‫And the second idea was what gradient descent is.
‫Mathematically we will merge these two and understand how gradient descent is helping us achieve the
‫minima and neural networks.