In this lecture we are going to derive the categorical cross entropy loss function which is what we
use when we are doing multi class classification.
At this point you probably already know that the categorical cross entropy is the negative log likelihood
for some distribution.
So let's jump right in and try to figure out what this distribution is.
In the case where we only have binary outcomes the correct discrete distribution is the Bernoulli distribution.
So what is the correct distribution for multiple categorical outcomes.
Well unlike the Gaussian and Bernoulli distributions this distribution is not named after a person.
It is simply called quite appropriately the categorical distribution
the usual analogy for the categorical distribution is a die roll.
So when you roll a die.
There are six possible outcomes.
Typically one two three all the way up to six.
Of course there are different kinds of dice.
Well you may have even more possible outcomes.
In general we will refer to the number of possible outcomes as K.
So what does the categorical distribution look like.
Well it looks a little strange as you can see it is a product each of the mew KS is a parameter telling
us the probability that the result of the dice roll was k.
In other words Mew K is equal to the probability that x is equal to k.
So what does this mean about the one function in the exponent.
This is what is called an indicator function.
It returns one and when its argument is true and 0 otherwise Knowing this it's easy to confirm that
if we plug in x equals 1 We should get that the probability that x is equal to 1 equals mu 1.
The probability that x is equal to 2 is Muthu and so forth.
In other words we can confirm that this is indeed the p MF for the categorical distribution
as usual we are going to pretend we've collected any data points and we are going to do maximum likelihood
estimation to find all the muse.
What is interesting about this likelihood is that because the likelihood is the product of P a maths
and the p MF already has a product we end up with two products in the likelihood also notice that there
are two indices in the exponent here.
There is the index eye and also the index K so that should remind you of a matrix because a matrix also
has two indices.
The row in the column
after taking the log.
We know that we get the log likelihood and we already know that the negative log likelihood is our categorical
cross entropy.
So if we take the negative of this expression we will get back something that looks exactly like the
categorical cross entropy
and thus we can relate this back to our categorical cross entropy loss function.
Notice here that it's a tiny bit different from our log likelihood and that we don't have an indicator
function anymore.
This is because an equivalent way of writing this is double indexing.
Why I k y k is called a 1 hot and quoted value because it can only take on the values 1 or 0 which row
can only have 1 1.
Remember that I represents the sample index and K represents the class label.
The class label can only be one value.
For example so for instance if I roll a die I can only get one number.
It must be six or five or four three it can't be six and three at the same time so as another example
if I have a picture and I'm classifying what type of animal it is it can only be one type of animal
if it's a picture of a cat.
It cannot simultaneously be a picture of a dog therefore only one of the K values can have a one representing
the target.
The rest of the values must be 0
you'll notice that one hot in cutting the targets like this can be quite inefficient.
Let's do a simple example.
Suppose we have four samples so and it's four and three classes cat dog or bird.
Suppose the targets are cat dog bird and cat.
So numerically we can represent this with the vector one two three one.
But in order to one hot and cold this vector we must turn it into a four by three matrix or an end by
K matrix containing only zeros and ones.
So the equivalent one hot encoding is 1 0 0 0 1 0 0 0 1 1 0 0.
Now suppose my prediction probabilities also an n by K matrix are as follows.
So for each sample and each class I have an associated probability it's the probability that I think
the input belongs to that class
after applying the categorical cross entropy formula.
We get this value for the cost.
Now you might be thinking that is a whole lot of multiplying by zero.
We already know that anything multiplied by 0 0.
So those values don't actually contribute anything to the cost.
The only values that do matter are the ones which match the target.
Remember the original target is the vector 1 2 3 1 What I would like to do ideally is to only take the
log of these highlighted values and then add them altogether.
The question is can I do that without one heart and calling the target first.
And indeed this is possible in number pi.
We can use the double indexing trick number higher res allow you to index using it such as lists and
other arrays instead of just single integers.
So if I wanted to take my predictions and index only the relevant values here's what I would do.
I would take y hat at equals 1 and K goes 1 I would take y hat at i equals 2 and K equals 2.
I would take y hat at i equals 3 and K close 3 and I would take my hat at I equals 4 and k equals 1
using the number pi double indexing method.
I can actually index all of these values at once I simply pass in all the row indices and other corresponding
column indices at the same time.
Importantly note that by doing this I never have to 1 hat and coat the targets I can pass them directly
into the square brackets as an exercise.
You should try this in actual num pi code to verify that it really works just note that in code or we
index starting at 0 instead of 1.
And so this is the idea behind the sparse categorical cross entropy.
So when you are in tensor flow 2.0 if you use the regular categorical cross entropy you are using the
one hot encoded target which requires n times K multiplications and additions.
If you use the sparse categorical cross entropy you were using the original target which requires only
n multiplications and additions.
Therefore it's usually better to use the original target and apply the spice categorical cross entropy
loss.