Hi and welcome to the lesson where we talk about the confusion matrix on classification report two very
important reports that help us analyze the performance of our classification model.
So like I said, a confusion matrix.
Basically, it's used on classification models where we have categorical outputs, not continuous value,
like a regression model, and it's often just performed on the test data set, although you can perform
it on the training dataset.
There's often less a less point to doing it on the training dataset.
You want to actually see an associate model's performance on unseen data, which is the test dataset,
and the confusion matrix is a very easily generated would say could lose function.
You can just import it like this.
And the two input arguments that this confusion matrix formula of function takes all the ground should
labels, which is your way test labels or why labels.
And these are test labels, I should say, and the predicted labels.
This is when you input the X test into your model and you get some predictions out of it.
So these are the two input arguments that the confusion matrix needs.
So this is a confusion matrix confusing, isn't it?
So what do these numbers mean?
Well, let's move on to the next slide, and I'll explain that to you.
So firstly, disregard this color bar here, but we'll get to that shortly.
What I want you to understand, firstly, is the accesses here.
So on the top here you have two predicted labels and on the right you have the true labels, true meaning
the ground with labels.
So why do we have a diagonal of large numbers going through this confusion matrix?
Well, it's quite simple, actually.
So let's take a look at the top left corner here.
Nine seven three.
What is this thing is that our model predicted a class zero nine hundred and seventy three times when
it actually was class zero.
So whenever we fitted digits that were actually zero, then the classifier.
The model predicted that it was zero nine hundred and seventy three times what is at times I mean instances,
because remember, our test dataset had 10000 numbers in it, images in it.
And if you divide 10000 by 10 roughly because it's not an evenly distributed class, you get roughly
a thousand or so numbers in each of each of them in each and each of the tests dataset.
So we have roughly a thousand zeros, a 2001 2002 trees and so on.
Roughly, it's not exactly tells them.
So what is this thing here?
This is what the model predicts.
The model predicted zero nine hundred and seventy three times when it was actually zero.
So what about these numbers here?
Just to this one, this one, this one's two?
Well, what I'm seeing here is that when the input was actually zero twice, the model incorrectly predicted
that it was a two, then sometimes repeated that zero was a five or six or seven twice in particular,
was it?
So this breakdown in this table helps us understand where our model is weak so you can see Caelynn.
That's why it is a diagonal in the middle, because one corresponds to this one here.
And to correspond to this one two here, that gives us this large diagonal in the middle, which is
desirable because that means your model is performing very well.
However, let's take a look at something here.
So let's look at it for it.
Idiot for here.
What else to following us is that five four five is here.
When the ground truth was five, it truly was a five hour model.
Got it right.
Eight hundred and eighty four times.
That's very good.
And in the red ball, this tells us that twice the model taught a five was a zero three times the model
to it, a five was a tree and so on for this.
So whoop, you give that.
This gives you a good understanding of how to understand the confusion matrix.
It's actually quite simple.
So if you if you're confused with this, just go over this lesson one more time, and hopefully it will
all make sense.
So let's take a look at this.
What mean issues can we deduce from a model's performance?
Immediately, what you should do is analyze look at a larger values that aren't in a diagonal to large
values.
Here are this nine and the seven.
What is this telling us here is that our model has a problem with thinking that is actually nice because
we know that this this rule here, all of this, all of these images here were actually false, but
nine times, which is not that much to be fair.
Nine times or model toward that four was nine and look so good the seven here now.
So this means that for the input.
Image I was seven, we got to 2014 that it was right, but seven times on one thought a seven was a
two.
So that's what this summary is here.
So again, I hope this helps you understand how to how to analyze performance, how to find weaknesses
of a model.
Try to figure out what your model is good at and weak at.
And maybe we need to strengthen the model.
So before we move ahead with some more detailed metrics, let's take a look at a simple binary classification
problem in this problem.
We have a classifier that predicts whether a patient has a disease based on an X-ray scan, so immediately
with they are four possible outcomes here.
These four possible outcomes are true negatives, false positives, false negatives and true positives.
So this is a confusion matrix here, actually for binary classification problem.
And as you can see, we have the brewers here with what the model predicted predicted.
No predicted ness.
Yes.
And then we have the true ground with answers here.
New yes.
So true negatives.
What do true negatives mean?
True negatives mean that a model correctly predicted new and the patient didn't have the disease?
Now what about false negatives?
Well, that's pretty bad in this case.
That means the patients actually had the diseases, but a model predicted no.
Now, in this case, the model predicted five times, which are false positives that the patient had
the disease.
However, in fact, he didn't wish he or she didn't.
And also four true positives, which is a good outcome, means that the model is working well with detecting
whether a person has the disease right here.
90 So immediately now.
What can we deduce from this?
Now, just to just to be clear, just so I don't confuse you these totals here.
Total true negatives is just some sum of these here for the true label total true.
These are basically how much patients had the disease, how much didn't have the disease and what the
model was predicting.
So the model predicted 50 people didn't have the disease and ninety five people had the disease.
And this is the reality here.
So let's take a look at accuracy.
How do we determine accuracy in this case here?
Well, actuaries accuracy is given simply by true positives, plus true negatives.
That's 40 plus 90 divided by the sample size, which is 145, which gives us 89 percent.
So based on this, we can see over accuracy for this model, which is a good term to use to baseline
in models.
A good metric.
However, it doesn't give us all the information as we have seen before, but our accuracy in this case
is 89 percent or 0.89.
Seven.
So what about a misclassification rate or error rate?
That's simply one minus the accuracy, which is basically 10 percent or zero point one zero three.
Now, let's take a look at some other metrics.
So the first one we'll talk about is true positive rate, also called sensitivity and most often called
recall.
So recall is a very important metric.
It basically tells us when it's a yes, meaning that when the ground truth is yes, how often are we
predicting yes?
It's a very important metric, especially with the disease case in that you don't want to miss a disease.
So when the person has the disease, how often are we predicting that the person has it that that disease?
And in this case, we can just derived from our stats here so we can see that we predicted the model
predicted that 90 times that the patient had the disease and when in fact in reality they were 100 patients
had the disease.
So we missed 10 percent.
So but it's simple to look it out.
It's just true positives over the true positive levels, which are these which is a hundred rate, our
total true positives.
So that's what we get this point.
Nine.
Now let's take a look at false positive rate.
Well, false positive rate again is quite simple.
It's when it's a new, basically.
So when a patient doesn't have the disease, how often are we predicting?
Yes.
So basically, this tells us whole.
In a way, it gives us an indication of how many times when it returns a yes.
Does it actually mean that a patient has the disease?
Well, the false positive rate is 11 percent, so that means 11 percent of our positives or predictions
of yes are actually no.
So if this was a very high number, you wouldn't know that this isn't a very reliable test for finding
that easiest disease.
Now let's talk about specificity or the true negative rate.
And this is the formula for it.
It's true negatives divided by the true negative label total number of true negative labels.
So in layman's terms, what is this telling us is that when we predict a new is it actually in know?
How often?
Is it actually A.?
So it's a very important case where if you if you're if you're being tested, OK, imagine the scenario.
You're going for COVID tests and you get a new but you want to know what is the probability of that
new being A..
Well, that's basically what specificity measures and in this case is point in.
Nine.
So as you can see, these are very important metrics when evaluating performances of classification
models.
Now let's take a look at precision.
Precision is another metric it's given by true positives, divided by how many times you predicted yes.
So what is this telling us here is that when it's a yes, how often is it right?
So when we predict 90 times that a patient has the disease and we predicted 95 positives in total that
tells us basically that we have ninety five percent to four yeses.
All right.
So it's a very good metric again to look at.
So this is a summary of some of these metrics.
Precision recall, and you may see another one here, a new one called F1.
Well, before we recap what recall and precision and talk about the trigger.
So let me start with what F1 is.
F1 is basically harmonic mean of what precision and recall.
So it's a way to encapsulate both precision and recall, which are two very important metrics in one.
And this is what with the formula, how it's worked out, how it's calculated.
So it's a basically overall score that takes in consideration both recoil and precision.
So just to recap, what recall is what a true positive rate or sensitivity.
Basically, it's saying when and say yes, how often do do we predict yes?
So in other words, how good of a how good is our classifier in identifying occurrences of our class
or finding the true positives in the class?
Precision is when we predict for years how often that, yes, actually correct.
So the trade offs here, sometimes we can accept a lower position with high recoil.
Let's take a look at disease prediction.
Let's let's look at COVID 19 testing.
We want to identify persons with the disease as much as possible at the expense of false positives.
What this means here is that we we don't want to miss people with COVID 19 because if we if we if they
have the disease yet they go to negative tests, they go back out in public and they can spread the
virus around.
So in this case, we we know false positives are relatively acceptable because of the cost of having
missed a person with the disease and giving them a false.
So what this means is that we will have a high recall with a low precision in this case.
So now let's take a look at classification reports.
You can see a classification report.
Basically, it gives us the precision recall, F1 score and support.
So what is simply the occurrences of each class?
It's not that important in terms of as a metric, it's a small class count, and it basically gives
us that for each class.
Here you can see.
So you can see for amnestied digit classifier, we're getting very good schools in precision recall
and F1 support.
So this is a quite good.
However, in reality, when a lot of Real-World data sets, it's not going to be this nice.
You're going to have some classes performing quite poorly, some performing much better.
So this is a very good classification report, is a very good tool in a matrix that you can use to analyze
this performance, especially when it went in terms of looking at precision and recall specific things.
So no, let's go ahead and trino amnesty and model that we've done before.
But now we're going to generate a classification report and a confusion matrix using Bord Keras and
PyTorch.
So I'll see you in the next chord lessons.
Thank you.