In this lecture we are going to discuss the binary cross entropy loss which is the correct lost function
to use when we are doing binary classification.
Let's recap the actual lost function and then we will use the rest of his lecture to determine how to
arrive at this lost function.
Since we know that this is going to be based in probability let's think about the associated probability
problem.
We know that the outcomes the Y's must be binary.
That means the targets in a binary classification problem must be 0 or 1 spam or not spam fraud or not
fraud etc..
The next question is what distribution is used for binary events.
The answer is the Bernoulli distribution
probably the most typical example of the Bernoulli distribution is the coin toss.
So imagine if you tossed a coin a bunch of times and you want to calculate the probability of heads.
Call that Mew.
Intuitively we know that MU is the number of heads divided by the total number of coin tosses.
Again we are going to go through the process of maximum likelihood estimation to see how we can arrive
at this answer.
All right.
So let's say we've tossed our coin a bunch of times.
Let's call the results x1 x 2 all the way up to X N.
In this case the IS can only take on it 2 values 0 or 1.
Let's say 1 means heads and zero means tails in order to calculate the likelihood we need the Bernoulli
P MF which is the equation you see here now there's one small difference between this example in the
previous example on the squared error.
The difference is that for the previous example we were working with heights which are continuous valued.
Now we are working with coin tosses which are discrete a coin toss can only give you 0 or 1.
It can't give you say zero point five zero point nine.
So we know from my study of probability that four continuous distributions we use the probability density
function but for discrete distributions we use the probability mass function or P MF.
So this is a P MF and it returns a probability rather than a probability density
since you already know the steps.
Let's get right to it.
We create our likelihood function which is the product of the P maps for each of the axes we've collected.
We know that we want to maximize the likelihood but the step we have to do before taking the derivative
is that we want to find the log likelihood at this point this equation should already look very familiar.
It's in the exact same form as the binary cross entropy except it negated
as usual if you were to take the derivative of the log likelihood set it to zero and solve for me you.
This is what you would get.
Curiously we arrive at the same answer as the guessing case the sample mean of x.
How can this be.
Why isn't it.
Number of has divided by n.
In actuality it is.
Remember that X can only take on the values 1 and 0.
So when x is 0 that's tails.
So they don't contribute anything to the sum of X the sum of X then is just the sum of a bunch of ones
which is the number of times we flipped heads.
So in fact this is exactly the same as number of heads divided by n
as before.
If we take the negative log likelihood and we put it side by side with the binary cross entropy we can
see the similarities.
We can conclude that what the binary cross entropy is really saying is that y is the result of a coin
toss with a probability of heads for that coin toss is given by y hat.
I analogous Lee x y is the result of a coin toss where MU is the probability of heads for that coin
toss
as a side note.
You want to keep in mind that we also usually divide the sum of the errors by N so that we get the average
binary cross entropy per data point.
This makes it just like the mean squared error in that the value is invariant to the number of samples
we have.
You can imagine that if we have lots of samples and we only use the sum the error would be very large
only due to the fact that we have a large number of samples in order to make the value of the error
more meaningful.
We can take the average and that makes our error more interpretable
So to conclude what have we done in this lecture.
We have shown that just like the mean squared error the binary cross entropy loss function is based
in probability.
What it says is that for the regression case where we use the means squared error it arises from the
Gaussian distribution for the binary classification case where we use the binary cross entropy it arises
from the Bernoulli distribution.
Importantly the common pattern between the two is that in both cases the error function is really just
the negative log likelihood and therefore the solution we end up finding in the end is called the maximum
likelihood solution.