In this lecture we are going to discuss another very important topic multi class classification previously
we discuss the binary classification which used a sigmoid at the output.
We also had signal for the hidden layer activation functions but we just got rid of that in favor of
the real you which tends to work better for the output.
However if we are doing binary classification the sigmoid is still the right choice binary classification
has many applications.
We can predict disease or notices is the user fraudulent or not.
Well the user click my ad or not.
Will the user accept a friend request or not.
And so on and so forth.
But there are situations that binary classification cannot handle such as when we might have multiple
categorical outcomes
let's think of some examples.
One example is optical character recognition or handwriting recognition.
If we want to recognize alphanumeric characters then there are at least 36 possibilities 10 digits and
26 letters.
What about speech recognition.
If we want to recognize words then each word counts as a separate category image classification is another
one and for a long time this was the standard benchmark which proved that deep learning works better
than other machine learning models.
You may have heard of a famous dataset called the image in that dataset which is made up of millions
of images in the thousand categories such as bagel vending machine and soccer ball.
Every year there is a contest to see who can build the best image that classifier and of course these
are all deep learning models.
So how does multi class classification work.
Well suppose we have just calculated the activation at the final layer of our neural network but have
not yet applied the activation function.
So we have a superscript l equal to w superscript l transpose dot Zi superscript l minus 1.
So that's the Z at the previous layer plus b superscript l it's important to remember that if the layer
size is k then a superscript l is a vector of size k what we would like to do is map these values to
probabilities for each of the K possible output categories.
Okay so that's your first lesson here.
If we have K possible outcomes then we should have K output nodes.
The question now is what function can actually map the vector a superscript l 2 probability values
let's think about what the requirements are for these probabilities.
We need a probability distribution over a K distinct values.
As a side note we assign these values the integers 0 to K minus 1 first since they are probabilities.
They must be non negative.
In other words they must be greater than or equal to zero.
But there is also an upper limit.
The maximum value of a probability is 1.
But more importantly for this to be a proper probability distribution all the possible different values
must sum to 1.
So the second requirement is that all the probabilities must sum to 1
luckily there is a function that accomplishes exactly this.
It is called the soft max function first let's drop the superscript l on the activation a for now so
we can just say a is a vector of size k then the soft max of this vector A is the exponential of a divided
by the sum of the exponential of a across each component.
As you can see this satisfies both of our requirements.
The exponential of any number is always positive so therefore each probability must be non negative.
Second because the denominator is just the sum of all the possible values of the numerator.
Then if we sum all the possible values of the numerator we just get the same sum and therefore we get
the same sum on the top and the bottom and they cancel out to give us 1
now that we know how to solve Max works.
Let's answer the most important question how do we use it in PI talk.
Well just like most other functions we've applied so far it's very simple.
There is an object we instantiate called soft Max.
In other words the only requirement is that you spell it correctly.
Of course you can always implement the soft Max yourself but in PI torch there's no need.
As a side note the south Max is considered an activation function but unlike the real you sigmoid and
tan h it is not meant for a hidden layer activations if you want to try using the soft Max as a hit
and their activation you are most welcome to but you'll generally find that it doesn't work that well.
So the soft Max is technically an activation function but we normally only use it when we're trying
to get an output probability from a vector of activation values such as at the end of a neural network
now that you know how to use the soft Maxim pi torch.
The next thing I'm going to tell you is to not use it just like we learned earlier with the sigmoid
and the binary cross entropy pi two which combines those two functions into a single loss for numerical
stability.
So pi talk is going to combine the soft Max function with the cross entropy loss into this single loss
which is simply called cross entropy loss pi talk does in fact have a standalone categorical loss which
is called an L L loss but that is outside the scope of this course.
In the example below we've implemented logistic regression but instead of being a binary logistic regression
it's now a multi class logistic regression that predicts k different categories.
As you can see the model is still just a basic linear layer.
All the magic really happens inside the cross entropy loss object.
The last thing I want to do in this lecture is summarize the three types of tasks we've encountered
so far that none that waste can perform and their corresponding activation functions two of them you
already saw in the previous section of this course.
And now we have a third.
So first we have a regression where we use it no activation function at all.
Sometimes we also call this the identity function which is just a fancy way of saying a function that
returns its input.
Second we have binary classification where we use the sigmoid activation function using the sigmoid.
We only need one output node because since we know the probability of the output being one the probability
of the output being 0 is just 1 minus that.
This is the same type of thing you see when you're looking at the Bernoulli distribution versus the
categorical distribution the Bernoulli distribution is a categorical distribution where there can only
be two outcomes.
Thus it only has one parameter.
The categorical distribution on the other hand if it has K outcomes it also has K parameters.
And that brings us to our third scenario which is multi class classification.
For this we use the soft Max activation function
one important thing to mention that sometimes confuses people is that these three activation functions
apply whether you are using a feed for neural network a linear model or even a more complicated network
like a CNN or an aunt.
It doesn't matter what kind of model you have these three activation functions always correspond to
these three tasks.
So for example if I don't want to use a neural network but just a simple linear classifier for multiple
classes then I would have just a single dense layer with a soft Max activation.
This is still called logistic regression by the way.
It's just not binary logistic regression.
Instead it's called multi class logistic regression it's important to remember that what came before
the final layer in the model is not related to the task.
You can have a CNN or an aunt in or any other kind of complicated neural network.
It is still the case that the final layer of the network will be a dense layer with one of these three
activation functions corresponding to the task being done.
Another important thing to remember is that the soft max function is more general whereas the sigmoid
can only handle binary classification.
The soft max function can handle multi class classification which includes binary classification.
So it's possible if you want to just use the soft Max for all types of classification.
If you set k equals 2 then you're doing binary classification.
It's just that in this scenario your outputs are a bit redundant because you already know that they
both sum to one and hence only one of them is really needed.