In this lecture, we are going to discuss another very important topic, multiclass classification.
Previously, we discussed binary classification, which used a sigmoid at the output.
We also had segments with a hidden layer activation functions, but we just got rid of that in favor
of the you, which tends to work better for the output.
However, if we are doing binary classification, the sigmoid is still the right choice.
Binary classification has many applications.
We can predict disease or notices.
Is the user fraudulent or not?
Will the user click my add or not?
Will the user accept a friend request or not?
And so on and so forth.
But there are situations that binary classification cannot handle, such as when we might have multiple
categorical outcomes.
Let's think of some examples, one example is optical character recognition or handwriting recognition,
if we want to recognize alphanumeric characters, then there are at least 36 possibilities, 10 digits
and twenty six letters.
What about speech recognition?
If we want to recognize words, then if word counts as a separate category.
Image classification is another one.
And for a long time, this was the standard benchmark which proved that deep learning worked better
than other machine learning models.
You may have heard of a famous data set called The Image in that data set, which is made up of millions
of images and a thousand categories, such as bagel vending machine and soccer ball.
Every year there is a contest to see who can build the best image that classifier.
And of course, these are all deep learning models.
So how does multiclass classification work?
Well, suppose we have just calculated the activation at the final layer of our neural network but have
not yet applied the activation function.
So we have a superscripts l equal to W superscript L transpose dot Z superscript L minus one.
So that's the Z at the previous layer plus B, superscript L.
It's important to remember that if the laicised is K, then a superscripts L is a vector of size K.
What we would like to do is map these values to probabilities for each of the key possible output categories.
OK, so that's your first lesson here.
If we have K possible outcomes then we should have a K output nodes.
The question now is what function can actually map the vector a superscript L to probability values.
Let's think about what the requirements are for these probabilities.
We need a probability distribution over kay distinct values.
As a side note, we assign these values the integers zero to K minus one first.
Since they are probabilities, they must be non-negative.
In other words, they must be greater than or equal to zero.
But there is also an upper limit.
The maximum value of a probability is one.
But more importantly, for this to be a proper probability distribution, all the possible different
values must sum to one.
So the second requirement is that all the probabilities must sum to one.
Luckily, there is a function that accomplishes exactly this, it is called the softmax function.
First, let's drop the superscript L on the activation A for now so we can just say A is a vector of
size.
K, then the softmax of this vector A is the exponential of a divided by the sum of the exponential
of across each component.
As you can see, this satisfies both of our requirements, the exponential of any number is always positive,
so therefore each probability must be non-negative.
Second, because the denominator is just the sum of all the possible values of the numerator.
Then if we sum all the possible values of the numerator, we just get the same sum and therefore we
get the same sum on the top in the bottom and they cancel out to give us one.
Now that we know how the softmax works, let's answer the most important question how do we use it intensive
flow?
Well, just like most of the other functions we've applied so far, it's very simple.
We just pass in the string softmax.
In other words, the only requirement is that you spell it correctly.
Of course, you can always implement the softmax yourself, but intenser flow, there's no need.
As a side note, the softmax is considered an activation function.
But unlike the real you sigmoid and Tanach, it is not meant for a hidden layer activations.
If you want to try using the softmax as a hidden layer activation, you are most welcome to.
But you'll generally find that it doesn't work that well.
So the softmax is technically an activation function, but we normally only use it when we're trying
to get an output probability from a vector of activation values such as at the end of a neural network.
The last thing I want to do in this lecture is summarize the three types of tasks we've encountered
so far that known that we can perform and their corresponding activation functions.
Two of them you already saw in the previous section of this course, and now we have a third.
So first we have regression where we use it, no activation function at all.
Sometimes we also call this the identity function, which is just a fancy way of saying a function that
returns its input.
Second, we have binary classification where we use the sigmoid activation function using the sigmoid.
We only need one output node because since we know the probability of the output being one, the probability
of the output being zero is just one minus that.
This is the same type of thing you see when you're looking at the Bernoulli distribution versus the
categorical distribution.
The Bernoulli distribution is a categorical distribution where there can only be two outcomes, thus
it only has one parameter the categorical distribution.
On the other hand, if it has key outcomes, it also has key parameters.
And that brings us to our third scenario, which is multiclass classification.
For this, we use the softmax activation function.
One important thing to mention that sometimes confuses people is that these three activation functions
apply, whether you are using a feed for neural network, a linear model or even a more complicated
network like CNN or an aunt, it it doesn't matter what kind of model you have.
These three activation functions always correspond to these three tasks.
So, for example, if I don't want to use a neural network, but just a simple linear classifier for
multiple classes, then I would have just a single dense layer with a softmax activation.
This is still called logistic regression.
By the way, it's just not binary logistic regression.
Instead, it's called multiclass logistic regression.
It's important to remember that what came before the final layer in the model is not related to the
task.
You can have a CNN or an on in or any other kind of complicated neural network.
It is still the case that the final layer of the network will be a dense layer with one of these three
activation functions corresponding to the task being done.
Another important thing to remember is that the softmax function is more general, whereas the sigmoid
can only handle binary classification.
The softmax function can handle multiclass classification, which includes binary classification.
So it's possible if you want to just use the softmax for all types of classification, if you set K
equals two, then you're doing binary classification.
It's just that in this scenario, your outputs are a bit redundant because you already know that they
both come to one and hence only one of them is really needed.