In this lecture, we are going to discuss another very important topic multiclass classification.
Previously, we discussed the binary classification, which used a sigmoid at the output.
We also had segments for the hidden layer activation functions.
But we just got rid of that in favor of the réélu, which tends to work better for the output.
However, if we are doing binary classification, the sigmoid is still the right choice.
Binary classification has many applications.
We can predict disease or notices.
Is the user fraudulent or not?
Well, the user click my ad or not.
Will the user accept a friend request or not?
And so on and so forth?
But there are situations that binary classification cannot handle, such as when we might have multiple
categorical outcomes.
Let's think of some examples.
One example is optical character recognition or handwriting recognition if we want to recognize alphanumeric
characters.
Then there are at least 36 possibilities, 10 digits and 26 letters.
What about speech recognition if we want to recognize words?
Then each word counts as a separate category.
Image classification is another one.
And for a long time, this was the standard benchmark which proved that deep learning worked better
than other machine learning models.
You may have heard of a famous dataset called the image in that dataset, which is made up of millions
of images and a thousand categories such as bagel vending machine and soccer ball.
Every year, there is a contest to see who can build the best image that classifier.
And of course, these are all deep learning models.
So how does multiclass classification work?
Well, suppose we have just calculated the activation at the final layer of our neural network, but
have not yet applied the activation function.
So we have a superscript l equal to W Superscript L transpose Dot Z superscript l minus one.
So that's the Z at the previous layer, plus b superscript l.
It's important to remember that if the layer size is K, then a superscript l is a vector of size K,
what we would like to do is map these values to probabilities for each of the K possible output categories.
OK, so that's your first lesson here.
If we have K possible outcomes, then we should have a K output nodes.
The question now is what function can actually map the vector a superscript l to probability values?
Let's think about what the requirements are for these probabilities.
We need a probability distribution over distinct values as a signal.
We assign these values the integers zero to K minus one first.
Since they are probabilities, they must be non negative.
In other words, they must be greater than or equal to zero.
But there is also an upper limit.
The maximum value of a probability is one.
But more importantly, for this to be a proper probability distribution, all the possible different
values must sum to one.
So the second requirement is that all the probabilities must sum to one.
Luckily, there is a function that accomplishes exactly this.
It is called the soft max function.
First, let's drop the superscript l on the activation A for now, so we can just say A is a vector
of size k.
Then the soft max of this vector A is the exponential of a divided by the sum of the exponential avai
across each component.
As you can see, this satisfies both of our requirements.
The exponential of any number is always positive.
So therefore, each probability must be not negative.
Second, because the denominator is just the sum of all the possible values of the numerator.
Then if we sum all the possible values of the numerator, we just get the same sum and therefore we
get the same sum on the top and the bottom, and they cancel out to give us one.
Now that we know how to soft max works, let's answer the most important question.
How do we use it in TensorFlow?
Well, just like most of the other functions we've applied so far, it's very simple.
We just pass in the strings of Max.
In other words, the only requirement is that you spell it correctly.
Of course, you can always implement the soft max yourself.
But in TensorFlow, there's no need.
As a side note, the SoC Max is considered an activation function.
But unlike the real you sigmoid and 10h, it is not meant for a hidden layer activations.
If you want to try using the soft max as a hidden layer activation, you are most welcome to, but you'll
generally find that it doesn't work that well.
So the soft max is technically in activation function, but we normally only use it when we're trying
to get an output probability from a vector of activation values, such as at the end of a neural network.
The last thing I want to do in this lecture is summarize the three types of tasks we've encountered
so far that none that was can perform and their corresponding activation functions.
Two of them you already saw in the previous section of this course, and now we have a third.
So first we have regression where we use no activation function at all.
Sometimes we also call this the identity function, which is just a fancy way of saying a function that
returns its input.
Second, we have binary classification where we use the sigmoid activation function.
Using the sigmoid.
We only need one output node because since we know the probability of the output being one, the probability
of the output being zero is just one minus that.
This is the same type of thing you see when you're looking at the Brinley distribution versus the categorical
distribution.
The Brinley distribution is a categorical distribution where there can only be two outcomes.
Thus, it only has one parameter.
The categorical distribution, on the other hand, if it has key outcomes, it also has key parameters.
And that brings us to our third scenario, which is multiclass classification.
For this, we use the soft max activation function.
One important thing to mention that sometimes confuses people is that these three activation functions
apply whether you are using a feedforward neural network, a linear model or even a more complicated
network like a CNN or an aunt.
It doesn't matter what kind of model you have, these three activation functions always correspond to
these three tasks.
So, for example, if I don't want to use a neural network but just a simple linear classifier for multiple
classes, then I would have just a single dense layer with a soft max activation.
This is still called logistic regression, by the way.
It's just not binary logistic regression.
Instead, it's called multiclass logistic regression.
It's important to remember that what came before the final layer in the model is not related to the
task.
You can have a CNN or an Aunt N or any other kind of complicated neural network.
It is still the case that the final layer of the network will be a dense layer with one of these three
activation functions corresponding to the task being done.
Another important thing to remember is that the South Max function is more general, whereas the sigmoid
can only handle binary classification.
The South Max function can handle multiclass classification, which includes binary classification.
So it's possible if you want to just use the South Max for all types of classification.
If you set K equals two, then you're doing binary classification.
It's just that in this scenario, your outputs are a bit redundant because you already know that they
both sum to one and hence only one of them is really needed.