‫Hey everyone.
‫Congratulations first of all.
‫If you understood whatever I taught you in the last few lectures you understand neural networks in the
‫way they are used for prediction purposes.
‫In this lecture I'm going to talk about several subtropics these sub topics came up in our lectures
‫previously also but to maintain the flow.
‫I did not discuss them in detail at that time.
‫Also this extra knowledge is often what is asked in interview questions.
‫In fact I'm going to cover this lecture in a question on the format so keeping your attention as this
‫is also very important.
‫The first off topic is of activation functions and the first question that we are going to discuss is
‫why do we use activation functions.
‫Let's see if we do not have any activation function.
‫What is the output of a neuron.
‫The output would be given by this equation W1 X1 plus W2 x2 plus B is equal to Z which is the output.
‫This means that the output could be any real number with no boundaries if you are solving for a regression
‫problem then this may be acceptable.
‫But when we are doing classification that is we want an output of yes no type or 1 0 type.
‫We need to read the output d to get a 0 1 paper output.
‫Also if there are only linear neurons in the whole network you can only predict a linear relationship
‫between input and output variables.
‫So the answer is we use an activation function for two reasons.
‫First to put boundary conditions on our output for classification.
‫The boundaries are obvious for regression also.
‫If you have some boundary you can use an activation function.
‫The second reason is to introduce nonlinear entity so that we can find complex nonlinear patterns.
‫Also
‫next question is what are the different types of activation functions earlier.
‫We had discussed two activation functions.
‫One was the step function which is zero below threshold value.
‫And then it suddenly jumps to one at differential value.
‫Then we discussed the sigmoid function which is a continuous S shape go with zero as the lower boundary
‫and one as the upper boundary for most practical business purposes sigmoid function is good enough but
‫for rare scenarios where computation is an issue we sometimes use these two other activation functions
‫also because of their convergence efficiency first is the hyperbolic tangent function or the PAN edge
‫the graph of this function is almost similar to the sigmoid in shape but it has different boundaries.
‫It has upper boundary of one at a lower boundary of minus one and because it is centered at zero it
‫almost always has better convergence efficiency than sigmoid.
‫The second is the loop which is short for identified linear unit.
‫It is very widely used function especially in the inner layers of regression neural networks.
‫This is how this function looks like till 0 the function also outputs 0 but after 0 function outputs
‫the same as input that is if X is equal to x.
‫So the lower one is 0 but there is no upper bound.
‫This function performs well because it is very easy to execute.
‫The reason for using this function and it in layers is that this function introduces nonlinear entity
‫different layers.
‫However on the output layer it is rarely used because for classification the right side of the function
‫is not bounded and therefore it cannot be used on the other hand for regression.
‫The left side of this function is bone and therefore this function cannot be used.
‫So this function is good for activating it in layers but not for activating output layers.
‫You can find a summary of all these activation functions in the next light.
‫This is for your reference.
‫This brings us to the next question which we have already answered can hit in layers and output layers
‫have different activation functions.
‫Answer is yes.
‫As I told you we can implement redo in the layers and the sigmoid in the output layer.
‫Any such combination is allowed by our software to
‫next question is what is multi class classification and is there any special activation function for
‫multi class classification.
‫So first of all what is multi class classification.
‫Suppose you are classifying into yes or no or 1 0 0.
‫This is binary classification because there are two classes only.
‫But if we have more than two classes.
‫So if we want to classify images into shirts trousers ties and socks.
‫Now the output cannot be 0 or 1.
‫We cannot do like this that we gave 0 for shirts 1 for trousers 2 4 days 3 for socks that would not
‫give us the right answer.
‫So we have to handle the situation in a little different manner for such a situation.
‫We have an activation function called soft Max
‫this activation function works similar to sigmoid.
‫Mark has an additional step.
‫So what we do in multi class classification is we usually keep as many output neurons as we have classes.
‫So if we have three classes like shirts trousers and socks we keep three neurons at the output layer.
‫You can see in this image these three output neurons correspond to three output classes these three
‫output neurons have these sigmoid activation function only.
‫So the output of first neuron would also lay between 0 and 1 and we can see that the output value is
‫corresponding to the probability of whether it is a shirt or not.
‫The second output would also be between 0 and 1 and we can see that it is the probability of whether
‫it is a browser or not.
‫And the third output will also be between 0 and 1 and we can see that it is the probability of whether
‫it is socks or not.
‫But the thing is that the item can be only one that is the sum of probabilities should come out to be
‫one.
‫Either it is shirt or it is trouser or it is socks.
‫To implement this we put an additional layer of soft Max and we input the desert of these three neurons
‫into this sort Max layer.
‫This soft Max layer just divides each of the value by the sum of all the values.
‫Not this output can be considered as the probability of that class occurring and the sum of all these
‫probabilities will also be equal to one
‫so for multi class classification a soft Max activation is often used on the output layer.
‫That's all about the activation functions.
‫The next topic is gradient descent.
‫The question I want to discuss here is what is the difference between gradient descent and stochastic
‫gradient descent.
‫This is a very common question in the mind of students because they find stochastic written at some
‫places and it is not written in some texts.
‫Let me clarify this for you.
‫What we discussed in our previous lectures was actually stochastic gradient descent because I told you
‫that we take each individual training record and update our weights and biases with each training record
‫when we run the whole forward and backward propagation for each individual training record that is stochastic
‫gradient descent.
‫But if you run forward propagation for entire training set at one go and find out the average error
‫on the entire set and then a blade DVDs and biases during backward propagation that is gradient descent.
‫There is another variation in which we make small batches out of the training set and use these batches
‫instead of completed.
‫This one is called mini bad gradient descent.
‫The point is stochastic gradient descent starts updating weights and biases rapidly but it finds difficulty
‫in converging whereas gradient descent is slow because in each pass it has to go through the entire
‫training set.
‫But the good thing about gradient descent is it converges very well so we have to accordingly select
‫which optimization technique is to be used.
‫We will see this further in the practical part of this course
‫lasting I want to discuss in this lecture is epoch in neural networks.
‫Epoch is one cycle through the full training data.
‫So when we say epoch is equal to five it means we want the full training data to be fed five things.
‫Note that epoch is different from iterations.
‫Suppose you have 1000 training records now.
‫If you start reading the records one by one like we do doing stochastic gradient descent you will have
‫to I created 1000 times
‫so I iterations are the number of times you execute any process within one full training set.
‫On the other hand if you have said epoch to do then this means that the 1000 training examples will
‫be fed two times either one by one or all at the same time or in many batches.
‫The idea of using epoch is to allow the network to rejig its performance on the same data.
‫We will see how we can specify and use epoch in our party collectives.
‫That's all in this lecture in the upcoming video.
‫We will summarize the key parameters that you must know while implementing neural networks in the software.