‫Hey, everyone, congratulations, first of all.
‫If you understood whatever I taught you in the last few lectures, you understand neural networks in
‫the way they are used for prediction purposes in this lecture.
‫I'm going to talk about several subtopics.
‫These topics came up in our lectures previously also. But to maintain the flow.
‫I did not discuss them in detail at that time.
‫Also, this extra knowledge is often what is asked in interview questions.
‫In fact, I'm going to cover this lecture in a question answer format.
‫So keeping attention as this is also very important.
‫The first subtopic is of activation functions.
‫And the first question that we are going to discuss is, why do we use activation functions?
‫Let's see if we do not have any activation function.
‫What is the output of a neuron, the output would be given by this equation, W1 X1 plus W2 X2
‫plus b is equal to Z, which is the output.
‫This means that the output could be any real number with no boundaries.
‫If you're solving for a regression problem, then this may be acceptable.
‫But when we are doing classification, that is, we want an output of yes no type or one zero type.
‫We need to treat the output z to get a zero one type of output.
‫Also, if there are only linear neurons in the whole network, you can only predict a linear relationship
‫between input and output variables.
‫So the answer is we use an activation function for two reasons.
‫First, to put boundary conditions on our output for classification.
‫The boundaries are obvious.
‫For regression, also, if you have some boundary, you can use an activation function.
‫The second reason is to introduce non-linearity so that we can find complex, nonlinear patterns.
‫Also.
‫Next question is, what are the different types of activation functions?
‫Earlier, we had discussed two activation functions.
‫One was the step function, which is zero till a threshold value.
‫And then it suddenly jumps to one at its actual value.
‫Then we discussed this sigmoid function, which is a continuous s shape curve with zero as the lowered
‫boundary.
‫And one as the upper boundary for most practical business purposes.
‫Sigmoid function is good enough, but for rare scenarios where competition is an issue, we sometimes
‫use these two other activation functions also because of their convergence efficiency.
‫First is the hyperbolic tangent function or the pan edge.
‫The graph of this function is almost similar to the sigmoid in shape, but it has different boundaries.
‫It has upper boundary of one at a lower boundary of minus one.
‫And because it is centered at zero, it almost always has better converges efficiency than sigmoid.
‫The second is Relu, which is Short for rectify linear unit.
‫It is very widely used function, especially in the hidden layers of regression neural networks.
‫This is how this function looks like, till zero
‫The function also outputs zero, but after zero function outputs the same as input.
‫That is, fx is equal to X.
‫So the lower bound is zero, but there is no upper bound.
‫This function performs well because it is very easy to execute.
‫The reason for using this function in hidden layers is that this function introduces non-linearity
‫in the hidden layers.
‫However, on the output layer, it is rarely used because for classification, the right side of the
‫function is not bounded and therefore it cannot be used.
‫On the other hand, for regression.
‫The left side of this function is bound and therefore this function cannot be used.
‫So this function is good for activating hidden layers.
‫But not for activating output layers.
‫You can find a summary of all these activation functions in the next slide.
‫This is for your reference.
‫This brings us to the next question, which we have already answered.
‫Can hidden layers and output layers have different activation functions.
‫Answer is yes.
‫As I told you, we can implement RELU in the hidden layers and a sigmoid in the output layer.
‫Any such combination is allowed by our software tool.
‫Next question is what is multiclass classification and is there any special activation function for
‫multiclass classification?
‫So first of all, what is multiclass classification.
‫Suppose you are classifying into
‫Yes or no.
‫Or one.
‫Or zero.
‫This is binary classification because there are two classes only.
‫But if we have more than two classes.
‫So if we want to classify images into shirts, trousers, ties and socks.
‫Now the output cannot be zero or one.
‫We cannot do like this that we give zero for shirts, one for trousers, two for ties, three for socks.
‫That would not give us the right answer.
‫So we have to handle the situation in a little different manner for such a situation.
‫We have an activation function called softmax.
‫This activation function works similar to sigmoid.
‫But has an additional step.
‫So what do we do in multiclass classification is we usually keep as many output neurons as we have classes.
‫So if we have three classes like shirts, trousers and socks, we keep three neurons at the output layer.
‫You can see in this image these three output neurons correspond to three output classes.
‫These three output neurons have the sigmoid activation function only.
‫So the output of first neuron would also lie between zero and one.
‫And we can say that the output value is corresponding to the probability of whether it is a shirt or
‫not.
‫The second output would also be between zero and one.
‫And we can say that it is the probability of whether it is a trouser or not.
‫And the third output will also be between zero and one.
‫And we can see that it is the probability of whether it is socks or not.
‫But the thing is that the item can be only one that is the sum of probabilities should come out to be
‫one.
‫Either it is shirt or it is trouser or it is socks to implement this.
‫We put an additional layer of softmax and we input the result of these three neurons into this soft
‫max layer.
‫This softmax layer just divides each of the value by the sum of all the values.
‫Now this output can be considered as the probability of that class occurring.
‫And the sum of all these probabilities will also be equal to one.
‫So for multiclass classification a softmax, Activision is often used on the output layer.
‫That's all about the activation functions.
‫The next topic is gradient descent.
‫The question I want to discuss here is what is the difference between gradient descent and stochastic
‫gradient descent?
‫This is a very common question in the mind of students because they find stochastic written at some
‫places.
‫And it is not written in some texts.
‫Let me clarify this for you.
‫What we discussed in our previous lectures was actually stochastic gradient descent, because I told
‫you that we take each individual training record and update our weights and biases.
‫With each training record, when we run the whole forward and backward propagation for each individual
‫training record, that is stochastic gradient descent.
‫But if you run forward propagation for entire training set at one, go and find out the average error
‫on the entire set.
‫And then update the weights and biases during backward propagation.
‫That is gradient descent.
‫There is another variation in which we make small batches out of the training set and use these batches
‫instead of complete set.
‫This one is called mini batch gradient descent
‫The point is, stochastic gradient descent starts updating weights and biases rapidly, but it finds
‫difficulty in converging.
‫Whereas gradient descent is slow because in each pass, it has to go through the entire training set.
‫But the good thing about gradient descent is it converges very well.
‫So we have to accordingly select which optimization technique is to be used.
‫We will see this further in the practical part of this course.
‫Last thing I want to discuss in this lecture is epoch in neural networks, epoch is one cycle through the
‫full training data.
‫So when we say epoch is equal to five, it means we want the full training data to be fed five times.
‫Note that epoch is different from iterations.
‫Suppose you have 1000 training records.
‫Now, if you start training the records one by one, like we do in stochastic gradient descent,
‫you will have to, iterate it 1000 times.
‫So iterations are the number of times you execute the process within one full training set.
‫On the other hand, if you have set Epoch to two, then this means that the 1000 training examples will
‫be fed two times, either one by one or all at the same time or in many batches.
‫The idea of using epoch is to allow the network to recheck its performance on the same data.
‫We will see how we can specify and use Epoch in a particular lectures.
‫That's all in this lecture, in the upcoming video.
‫We will summarize the key parameters that you must know while implementing neural networks in the software.