-: Hello and welcome back to the course on deep learning.
This is an additional tutorial to talk
about the Softmax and Cross Entropy functions.
It is not a hundred percent necessary in order
for you to go through all of the parts that we've been
through in the main part of this section where we're talking
about the convolutional neural networks.
But at the same time, I thought it would be a good addition
to your bag of knowledge and skill set.
So let's go ahead and dig into these functions.
So to start off with, what we have here
is the convolutional neural network that we built
in the main part of the section, and then
at the end it pops out some probabilities
0.95 for a dog and 0.055% for a cat.
Given that photo in the left as an input.
This is after the training has been conducted.
This is actually, it's running
and it's classifying a certain image.
And so the question here is how come these two values
add up to one?
Because as far as we know from everything that we've learned
about artificial neural networks, there is nothing to say
that these two final neurons are connected
between each other.
So how would they know what the value of-
how would each one of them know what
the value of the other one is
and how would they know to add their values up to one?
Well, the answer is they wouldn't
in the classic version of an artificial neural network.
And the only way that they do is because we
introduce a special function called the Softmax function
in order to help us out of this situation.
So normally what would happen is the dog
and the cat neurons would have any kind of real values.
They don't have to add up to one.
But then we would apply the Softmax function
which is written up over there at the top
and that would bring these values to be
between zero and one and it would make them add up to one.
And to quote Wikipedia, the Softmax function
or the normalized exponential function is a generalization
of the logistic function that
"squashes" a K dimensional vector of arbitrary real values
to a K dimensional vector of real values
in the range of zero to one that I'll add up to one.
So basically, it does exactly what we want.
It brings these values to be between zero and one
and makes sure that they add up to one.
And the way it works is that
the way that this is possible is that
because at the bottom over here
you can see that there's a summation.
So it takes the exponent and puts it
in the power of Z and adds it up.
So Z one, Z two across all of your classes
all of these values.
And so there
that's your normalization happening right there.
So that's how the Softmax function works.
And it makes sense to introduce the Softmax function
into convolutional neural networks because how
strange would it be if you had a possible classes of a dog
and a cat and for the dog class you had probability of 80%
and for the cat class you had a probability of 45%, right?
It just doesn't make sense like that.
And therefore it's much better when
you introduce the Softmax function
and that's what you'll find happening most of the time
in convolutional neural networks.
Now the other thing is that the Softmax function comes hand
in hand with something called the Cross Entropy function
and it's a very handy thing for us.
So let's first look at the formula.
This is what the Cross Entropy function looks like.
We're actually going to be using a different calculation.
We're gonna be using this representation
of the Cross Entropy, but the result's basically the same.
This is just easier to calculate.
And what I know this might sound very unrelated
to anything right now, just formulas on your screen
but there'll be some additional recommended reading
at the end of this section.
So don't worry if you're not picking up on the math
like if we haven't explained the math right now.
But the point here is that what is the Cross Entropy?
Well, a Cross Entropy function
remember how we previously in artificial neural networks
we had a function called the mean squared error function
which we used as the cost function
for assessing our network performance.
And our goal was to minimize the MSE
in order to optimize our network performance.
Well, that was our cross function then.
There and in convolutional neural networks
you can still use MSE
but a better option in convolutional neural networks
after you apply the Softmax function turns
out to be the Cross Entropy function.
And in convolutional neural networks
when you apply Cross Entropy function
it's not called the cost function anymore
it's called the loss function.
And they're very similar
and they're just little terminological differences
and like little, a bit different in what they mean.
But for our purposes, it's pretty much the same thing.
And what happens is the loss function is again
something that we want to minimize
in order to maximize the performance of our network.
So let's have a look at a quick example
of how this function can be applied.
So let's say we've put an image of a dog into our network.
The predicted value for dog 0.9
and this is during the training.
So we know the label that is a dog.
So the predicted value is 0.9
the predicted value for cat is 0.1.
Then here we have the label.
So we know it's a dog because this is training
and one for dog, zero for cat.
And so in this case, you need to use
you need to plug these numbers
into your formula for the Cross Entropy.
So how you would do it is the values
on the left go into the variable Q
the one that is under the logarithm on the right side
and the values from the right would go into P.
And so it's important to remember which one goes where
because if you get them wrong
you don't wanna be taking a logarithm
from a zero value or a logarithm from a one.
So you just want to plug them in, make sure you plug them
into the correct places, and then you basically add that up.
So that's how the Cross Entropy works.
And we'll look at a-
Actually right now we're just going to look at a
specific step by step example
of applying this function in real life.
And it'll kind of make more sense what Cross Entropy is.
And it'll be less...
My goal in this tutorial is to make you more comfortable
with Cross Entropy because it can sound very convoluted
and no pun intended.
It can, like convolutional neural networks
it can sound very complex, right? Scary.
But it's not, that's the point.
So let's go ahead and apply it
just so we know that it's not scary.
So here's neural net
and also this will explain why we're doing this
why we are looking at different cross functions.
So neural network one, neural network two
let's say we have two neural networks
and then we pass an image of a dog
and we know that this is a dog and not a cat.
And then we have another image
of a cat this time, an animal, and it's a cat, not a dog.
And here we have a weird looking animal
which is in fact a dog, not a cat if you look very closely.
So we wanna see what our neural networks will predict.
In the first case, neural network one
90% dog, 10% cat, correct.
Neural network number two, 60% dog, 40% cat.
Still correct, worse but correct.
Second option, first neural network
10% dog, 90% cat, correct.
Neural network number two, 30% dog, 70% cat.
Worse but still correct.
And then finally, neural network one
in image three neural, one 40% dog, 60% cat. Incorrect.
Neural network number two, 10% dog, 90% cat.
Incorrect and worse.
So the key here is that even though
both networks got it wrong in the last one
throughout all three images, neural network one
was outperforming neural network too.
So even in the last case it, it was very
it gave dog like a 40% chance as opposed to
neural network two only gave dog a 10% chance.
So neural network one is outperforming
across the board when compared to neural network two.
And so now we're going to look
at the functions that can measure performance
that we've kind of talked about already.
So let's put these into a table.
So there's neural network one, you have the row number
so that's the image number.
And then for image one, you have what it predicted
90% dog, 10% cat.
So those are the hat variables.
And then you have the actual values.
So dog, correct, cat incorrect.
Same thing for image number two
and same thing for image number three.
And same for neural network number two.
So dog, 60%, cat 40% in the first image
that's what it predicted.
Correct answer was dog, not cat and so on.
And so now let's see what errors we can actually get.
So what errors we can calculate to estimate the performance
and monitor the performance of our networks.
So one type of error is called the classification error.
And that is basically just asking
did you get it right or not?
Regardless of the probabilities, it's just
did you get it right or did you not get it right?
So in both cases, for both neural networks, each of them
they got one or, so this is how many they got wrong.
So they got one out of three wrong.
So 33% error rate for neural network one
and 33% error rate for neural network two.
And so basically from this standpoint
both neural networks perform at the same level
but we know that's not true.
We know that neural network one
is outperforming neural network two.
That's why a classification error is not a good measure
especially for the purposes of back propagation.
Means squared error different
and by the way, I did these calculations in Excel.
I just didn't want to bore you with them
but you can totally just sit down
and do them on a paper or in Excel.
These are very straightforward calculations.
Just basically take the sum of squared errors
and then just take the average across your
across your observations.
And that's pretty much it.
So for neural network one, you get 25%.
For neural network two, you get 71% error rate.
So as you can see, this one is more accurate.
It's telling us that neural network one
has a much lower error rate than neural network two.
And then Cross Entropy, again we've seen the formula.
You can also calculate this.
This is actually even easier to calculate than
the mean squared error. Cross Entropy gives you 38%
for neural network one and 1.06 for neural network two.
So you can see the results are a bit different when you look
at them like that, when you look at, you know
the mean squared error and Cross Entropy.
The question of why would you use Cross Entropy over
mean squared error isn't just about the kind
of numbers that they spit out.
These calculations were just to show you
that this is all, it's all doable.
You can just do it on a paper.
These are not very intense mathematics.
These are pretty simple, straightforward things.
But the question of why would you use mean Cross Entropy
over mean squared error?
It's a very, very good question to ask.
I'm glad you asked it.
The answer to that is like
there's several advantages
of Cross Entropy
over mean squared error which are not obvious.
And so I'll mention a couple, then I'll let you know
where you can find out more.
So one of them is that if for instance
at the very start of your back propagation
your output value is very, very, very, very tiny. Very tiny.
So it's much smaller than the actual value that you want.
Then at the very start, the gradient
in your gradient descent will be very, very low
and it won't be enough.
It'll be very hard for the neural network
to actually start doing something
and start moving around and start adjusting those weights
and start actually moving in the right direction.
Whereas when you use something like the Cross Entropy
because it's got that logarithm in it
it actually helps the network assess even a small error
like that and do something about it.
Here's how to think about it.
So let's say
again, this is a very intuitive approach.
There's gonna be a link to the mathematics
and you can derive these things
through the mathematics in more detail
but a very intuitive approach.
Let's say
your outcome that you want is one
and right now you are at one one-millionth of one, right?
0.000001.
And then you improve next time, you improve your outcome
from one-millionth to one-thousandth.
And in terms of if you calculate the squared error
you're just subtracting one from the other, or basically
in each case you're calculating the square error.
And you'll see that the squared errors
when you compare one case versus the other
it didn't change that much.
You didn't improve your network that much when
you're looking at the mean squared error.
But if you're looking at the Cross Entropy
because you're taking a logarithm and then
you are comparing the two, dividing one by the other.
You will see that you have actually improved
your network significantly.
So that jump from one-millionth to one-thousandth
in mean squared error terms will be very low.
It will be insignificant and
it won't guide your gradient boosting process
or your back propagation in the right direction.
It will guide it in the right direction
but it'll be like a very slow guidance.
It won't have enough power.
Whereas if you do it through Cross Entropy
Cross Entropy will understand that, oh, even though
these are very small adjustments that are just
you know, making a tiny change in absolute terms
in relative terms, it's a huge improvement
and we are definitely going in the right direction.
Let's keep going that way.
So Cross Entropy will help your neural network
get to the optimal state. It's a better way
for the neural network to get to an optimal state.
But bear in mind
that this only works when the Cross Entropy
is the preferred method only for classification.
So if you're talking about things like regression
like which we had in artificial neural networks
then you would rather go with mean squared error.
Whereas Cross Entropy is better for classification.
And again, it has to do with the fact
that we're using Softmax function.
So that's a kind of intuitive explanation of that.
A good place to learn a bit more about that
if you're really interested in why
are we using Cross Entropy versus mean squared error.
Google a video by Geoffrey Hinton called
the Softmax Output Function
and he explains it very well.
And, you know, being the godfather of deep learning
who can explain it better anyway.
And by the way, any video by Geoffrey Hinton is golden.
He's just got a huge talent for explaining things.
Anyway, so that's Softmax versus Cross Entropy.
I hope that gives you kind of
like an intuitive understanding of what's going on here.
But more importantly that you are not put off
by the term Cross Entropy
because Hadlan will mention it in the practical tutorials
and I wanted to make sure that you're prepared for that.
And it's just another way
of calculating your loss function
and another way of optimizing your network
which is specifically tailored to classification problems
and therefore convolutional neural networks
and comes hand in hand with the Softmax function.
So additional reading, if you'd like
a light introduction into Cross Entropy
if you're interested in Cross Entropy a bit more, of course
a good article to check out is called
A Friendly Introduction to Cross Entropy Loss
by Rob DiPietro, 2016.
Here's the link below. Very, very nice.
Very soft, no super complex math.
Good analogies, good examples.
He uses analogies of cars
and you're looking at cars and he talks about information
and bits and restrictions and you know
how would you encode this?
How would you encode that?
It's a good article to have a look at
and will give you a good overview of Cross Entropy
like from an introductory standpoint. If you want to dig
into the heavy math, like what you see here, then check
out an article by, or a blog by How to Implement
a Neural Network Intermezzo 2.
So intermezzo is like an intermediary thing
intermittent, you know, like when you go
to a theater and you have like a break between
the first part and the second part
because he's like going through all these steps
and then he says, Oh, I gotta explain this first.
And yeah, so that's why it's called Intermezzo
No other reason, as far as I understand.
The article is by Peter Rolands, 2016 as well.
So both are quite recent.
And yeah, check out this if you would like to dig into
the mathematics behind Cross Entropy
behind Softmax and Cross Entropy in this article, actually.
So there we go.
That's all there is to these two.
Hopefully I was able to add some additional clarity
and good luck with that.
It's gonna be fun and enjoy the practical tutorials.
I'll see you next time.
Until then, enjoy deep learning.