Hello, everyone, and welcome to this new section in which we look at other methods of evaluating our
model other than the binary accuracy which will be seen so far.
So in this section we'll look at how to compute the true positives, false positives, true negatives,
false negatives, the precision, the recall, the A under the curve, how to come up with conversion
metrics like this.
And finally, how to plot out an rosy curve like this one, which permits us select the threshold more
efficiently.
Don't forget to subscribe and hit the notification button so you never miss amazing content like this.
Let's now look at other ways of evaluating our model other than the accuracy which we've seen so far.
To better understand why working with all of the accuracy isn't always a great idea, we have to take
into consideration the fact that our model on a test set has 94% accuracy.
Now, this means that we have six out of 100 predictions, which are actually false.
Now, what if I get to the hospital and I'm told that I don't have malaria when in fact I actually have
this disease?
So that said, the model predicts an infected and I actually have the parasite in my bloodstream.
Now, this particular situation becomes very dangerous because the patient gets back home thinking he
or she doesn't need any treatment, whereas that patient actually has this parasite.
You see that even with a 94% accuracy, we wouldn't be able to save ourselves from such chaotic model
predictions.
Now, in another example, you have a situation where the actual is, let's put it out here.
So in another example, we can have a situation where the actual is on parasitized.
So actually you do not have this parasite, but the model predicts that you have the parasite.
Now, in this case, although we have a wrong prediction, we have actually a less chaotic situation
as compared to this previous case here since.
At least actually you are uninfected.
If we consider negative, that's this negative to be uninfected and positive to be passed to contain
the parasite.
That's P.
So yeah, we consider we have positive and then we have negative.
So here we have negative, uninfected and positive parasitized.
Now if we let this, you will find that this first situation where actually we have the parasites and
the model predicts on parasitized is known as a false negative.
So here we have this false negative.
And this is because the model predicts negative when it isn't actually negative.
So since we have this wrong prediction for negative, we call it a false negative.
And in this case, where we have the model predicting parasitized that is positive when it's actually
negative, we call this a false positive.
So, yeah, we have FP.
And here we have f DN.
Now, there are two other scenarios that is with c, n, and TP for the T, and we have the true negative,
the true negatives and the CP, the true positives for the t n We have the model predicting negative
when actually we are negative.
That is, we have the model saying this is uninfected when actually it's uninfected.
So that's a true negative and then for the true positive, the model predicts a positive that's parasitized
when actually it's parasitized.
So that's a true positive.
Now hopefully you've understood the concepts of true negatives, true positives, false negatives and
false positives.
We could then summarize all this information in this matrix known as the confusion matrix and this confusion
matrix.
We have the true negatives, the number of true negatives here, the number of true positives, the
number of false negatives and the number of false positives.
This means that if we have a test set of, say, 2750 different data points and then we run this or
we evaluate this with our model, we'll be able to get this number of true negatives, get this number
of false negatives, number of true positives, number of false positives, and hence better evaluate
this model.
So if we take this example where we've evaluated our model on the test set and then this model A produces
this coefficient matrix and this Model B produces other conversion matrix, where here we see we have
four the true negatives and true passes.
We have 1000, 1000 as 2000 correctly predicted data points.
And then here we have 1002 thousand, 2000 correctly predicted the points.
And then for this model A, we have 700 false positives and 50 false negatives, whereas for Model B
we have 50 false positives and 700 false negatives.
Recall we are defined negative to be uninfected and positive to be parasitized, and hence if we are
to choose a model between A and B, we'll try to choose that model which minimizes the number of false
negatives.
So we are not saying that we shouldn't minimize the number of false positives because we have to try
to minimize all the false predictions.
But then since we have false negatives, we are telling a sick person that he or she isn't sick.
This at least, is worse than telling a healthy person that he or she is sick.
And so we'll try to prioritize the number of false negatives.
And based on this prioritization, we are going to prefer Model A since we have the smaller number of
false negatives.
And so your would choose model A over model B.
Now, as a quick note, you may decide to say negative is for parasitized and positive is for uninfected.
It isn't a must that this must be tied together like this.
But for clarity, purposes is better to look at it this way since saying you are tested negative means
you are uninfected and test a positive means you are parasitized.
Now you should also note that depending on the kind of problem you want to solve, in some cases you
will want to prioritize minimizing the number of false positive over the number of false negatives.
So this actually depends on the problem you're trying to solve.
But in this case, we are prioritizing the number of false negatives.
Now, based on what we've seen so far, we're going to introduce several new performance metrics and
all this take up these different formulas.
We could see right here we have the precision, which is the not the number of true positives divided
by the number of true positives, plus the number of false positives recalled Trump.
True positives divided by a number of true positives, plus number of false negatives.
The accuracy, the number of true negatives, plus number of true positives, divided by a number of
true negatives plus true positives plus false negatives plus false positives.
So we'll stop for this first three for now.
Now what do you notice?
You'll notice that in this position, recall we have this true positive, true, positive.
And here true positive, true, positive.
What differentiates them is the fact that in the position we have false positive, the denominator and
the recall, we have false negative in the denominator.
This means that if the number of false negatives is high, that is, we have let's say we have a constant,
a constant K divided by a high value.
So constant divided by a high value, so constant, divided by high here we're going to have a low output.
And so if we want to have a low recall, then we need to have a high number of false negatives.
And if we want to have a low precision, then we need to have a high number of false positives.
Now, in our case, we're trying to minimize the number of false negatives.
And since we're trying to minimize the number of false negatives, it means that we're trying to maximize
the recall since minimizing this denominator will entail maximizing this overall TP on TP plus F and
and so here we're trying to prioritize the recall over the precision.
Now, if you look at the accuracy, you'll notice that we have ten plus.
TP And ten plus.
TP Right here, ten plus.
TP Ten plus.
TP And we have F and plus FP.
If you are keen enough, you should see that this accuracy doesn't give any priority for whether the
false negatives or the false positives.
It treats this two as the same.
But as we've seen previously in the real world, in solving real world problems, many times we would
have to prioritize.
Hence, the accuracy may not always be the best metrics for our problem.
In our case, we find that using the recall is even better than using the accuracy.
As with the recall, we get to reduce or with the recall, we get to see whether our model does well
at minimizing the number of false negatives.
Now we also have this F one car two times the precision times.
Recall the suppression of this recall divided by the precision, plus recall the specificity, the number
of true negatives, divided by number of true negatives, plus number of false positives.
And then we also have this RC plot right here, our assistance for receiver operating characteristics.
Here we have the true positive rate and the false positive rate.
The true positive rate is the number of true positive.
Double number of true positive plus number.
All false negatives, which happens to be the recall.
And then the false positive rate is a number of false positives divided by a number of false positives,
plus the number of true negatives, which if you look carefully, you'll find that it's equal one minus
the specificity which has been defined right here.
Before getting to understand this RC plot which we've put out here, let's recall this two models which
we had described previously, that's Model A and Model B, where Model A had a smaller number of false
negatives as compared to Model B.
Now, if we pick out just as Model A, and then we are interested in reducing this or let's say we pick
our Model B, suppose we pick our Model B, our interested in reducing this number of false negatives
right here.
Then one solution could be that of modifying the threshold.
So if we have a threshold of 0.5, meaning that a both like we have this 0.5 below 0.5, we consider
a negative above 0.5, we consider a positive that is parasitized and then below uninfected.
Then what I could do here is reduce this threshold.
So if I take this threshold to say a value of 0.2, let's take this here.
So I've reduced this threshold now to 0.2.
You'll see that for most of the predictions, our model is going to say that this is a parasitized output.
Since now this threshold has been reduced.
This means that if we have a model prediction of 0.3, which initially would have been uninfected,
now this Model C is this as parasitized.
And so this makes it now more difficult for our model to have false negatives, since our model now
has this tendency of predicting that given an input image is parasitized.
That said, we now look, we now need to look for a way that we could automate this process.
That is, we want to be able to choose this threshold correctly or rightly, because if, let's say
we take a threshold of say, 0.001, it means that anytime our model predicts less than 0.01, it's
an uninfected and then greater than 0.01 is parasitized, then this will be very dangerous for the overall
model performance as now most times we would have the model predicting that the input is parasitized.
And so our M here is to pick this threshold such that this number of true positives and true negatives
we've had right here don't get reduced.
Now, the way we could look at this is now by using this Rosie plot with this RC plot where you actually
have your is the different true positive rates and false positive rates you would have at a given threshold.
So this means that a point picture is just a given threshold.
Now let's suppose that the threshold 0.5 is about yours.
So let's pick this.
Let's suppose that this is 0.5.
We could pick another threshold.
Let's say this one is say 0.2 and so on and so forth.
Let's say this one is 0.1.
Now, we could have another model with this different Rosie plot, another one with this kind of Rosie
plot.
But note that overall, our aim is to ensure that this false positive rate is minimized and the true
positive rate is maximized.
So if we have an RC plot, which is like this so let's let's re draw this.
If we have this kind of Rosie plot that is one that goes up straight like this and then comes this way,
we have this right here.
So if we have this kind of Rosie plot, then we will be able to pick out this threshold right here,
0.6 We'll be able to pick this threshold because at this threshold for this threshold at this point,
the true positive rate is at as high as value that is of one.
And then the false positive rate is at its lowest possible value that is of zero.
So yeah, we have one and then zero.
So here we go, we have this and then we pick out this threshold.
Now this value of x could be five, could be four or whatever.
So we have zero point whatever value will lead us to this.
Nonetheless, many times we wouldn't have this kind of plots, so we'll do it plus, which will look
like this, this and so on and so forth.
Now, once given a plot like this one, let's suppose we have a plot like this one, the Emir two axis
of the right equation.
If I want to make sure that my recall is always maximized, which is our case, we will try to ensure
that we pick out this points around this.
So we try to pick out this points to the top right here.
Let's take some of this off.
So as we're saying, if we want to maximize our recall, is that normal?
That will pick out threshold values which will take us around this region because it's around this region
that our recall is maximized.
Let's do this.
So you could see that clear.
Okay.
Now, the problem with picking a point around this is that when you pick a point around this, the false
positive rates is increased.
So you need to find that balance between this false positive rate and true positive rate.
So it would be much logical to pick a point around this right here so we could pick around this region
instead.
Of this previous region right here.
Now, you can see that in this region.
Let's say we pick this point.
We pick this point.
Your false positive rate now is smaller.
While your true positive rate is or your recall is maximized, though it isn't the best possible recall
we could have.
But trying to focus on getting that recall of one will lead you into trouble.
Since getting a recall of one in this case will increase our false positive rate.
And then if we're dealing with a problem where we're trying to maximize the precision, then in that
case, we want to ensure that this false positive rate is minimized.
And so in this kind of problems, we want to pick a point around this.
So you see, we want to pick this kind of value since our lists are false, positive rate is minimized.
But then if you want to go and pick a point around this year, you would have a false positive rate
of zero.
But doing this will get you this into trouble because, yeah, you're going to have a true positive
rate, which is very small.
And so you need to get that balance.
Now, if you have a problem where it doesn't really matter that it's you, I've been trying to prioritize
the position or the recall and working with accuracy just fine, then you could pick out this point
right here.
So this is when you are having or you don't have to prioritize on any areas when you're trying to prioritize
on the recall and yours is when you're trying to prioritize on the precision.
But I just want grittiness with this tool that is the rosy plot.
We are able to pick out this point and then automatically get that threshold we need to work with.
And so when doing predictions, we will not or we may not use 0.5, but we're going to use a certain
threshold which will suit these objectives.
We've set initially.
Now we'll now move to the area under the curve for the area under the curve.
We generally use this when we compare comparing to models.
Here we have this model, let's call it model A, It's not it's not actually this Model A is a different
model A let's call this model alpha.
And then we have this auto model which we shall call model beta.
So we have model alpha and model beta.
It's clear that model beta is better because it gives us better options.
Now you'll see that if I find myself here, I get it better.
True positive rate false positive rate balance as compared to when I find myself at this position.
And so if we are comparing this to models, we could make use of the area under the curve by calculating
this area covered here.
So let's bound this.
And then for Alpha, we have this a area under the curve popularly known as a you see a small U and
then see which will give us this.
And then for beta, we're going to have this area under the curve now which covers this area, plus
this extra area right here.
And so in general, if we have two models and then we want to compare them, then we could use this
area under the curve, since it shows us how much freedom we have in playing around with this thresholds.
We now get back to the code and see how we're going to implement this new matrix we've just talked about.
So right here we have Matrix and then we have the false positives, we have the false negatives, we
have the true positives, we have the true negatives, we have the precision, we have the recall,
we have the AUC, AUC.
And then since we're dealing with a binary classification problem here, we have binary accuracy, we
run that, that's fine.
And then right here, instead of having this matrix, we define this matrix list, which will contain
the different matrix which we've just talked of.
So we have the true positive false positive right up to the AUC.
Let's run this compile and then feed our model.
You'll see that as we train we have the true positives, false positives, true negatives, false negatives,
accuracy, precision recall, and AUC scores which are being given to us.
As you could see, the other results we obtained, we're going to go ahead to evaluate our model.
So let's get to the model evaluation.
We run this test data and then we have our model evaluated.
There we go.
Our model has been evaluated.
As you could see, we have this.
Last 0.35.
Number of true positives 1323.
Number of false positives.
Number of true negatives.
False negatives.
Accuracy.
Precision recall and AUC.
So that's what we have for this model.