This video tutorial we will look at mean average precision.
So mean average precision is a metric that is used to evaluate the performance of object detection models
in computer vision.
For example, if I are fine tune the Yolov5 model on fire data set so that I can detect fire in images,
videos, or in a live webcam feed.
So after I have fine tuned the Yolov5 model on fire data set, and I now have a fire detection model,
how I can evaluate the performance of my fine tune model.
So to evaluate the performance of the fine tuned model, I will be calculating a metric that is mean
average precision.
So mean average precision is a commonly used metric to evaluate the performance of object detection
models.
Many object detection models such as popstar R-cnn, MobileNet SSD, and YOLO series which include Yolov5,
Yolov6, Yolov7, Yolo v eight, neuron V9 use mean average precision to evaluate their models.
So mean average precision formula is based on the following submatrix.
Or we can say that following are the building blocks of mean average precision, which include confusion
matrix, intersection over union, precision, and recall.
So to calculate the precision and recall values, we need to calculate true positives and false positives.
And to calculate the true positives and false positives, we need to have a confusion matrix and to
calculate true positives and false positives.
In confusion matrix, we need to, uh, get the intersection over union.
So these four things are quite interlinked with each other.
So the confusion matrix is based on the following four attributes, which are given over here, though
the confusion.
To calculate the confusion matrix, we need to calculate true positives.
False positives or false negatives and true negatives.
So what is true positives.
So for example I have a car in an image.
So my model predicted a label.
So my model predicted that there is a car inside this image and it matches correctly with my ground
truth.
So in my ground truth prediction or in my ground truth model annotation, I have, uh, annotated my,
uh, the image that there is a car and my model predicted that there is a car in the image as well.
So my model prediction matches with my ground truth as well.
So if my model prediction matches with the ground truth, then we can say that this is a true positive
prediction.
So now you can see that for example, uh, you can see that uh, to understand confusion matrix, let's
take an example of this classification problem.
Uh, a model has to identify whether there are hot dogs in the image.
The model can be either, uh, the prediction can be either correct or incorrect.
So for example, like you can see that, uh, my model predicted that there is a hot dog inside this
image.
Okay.
So this is a true positive because we can see that there is a hot dog inside this image okay.
And then next is true negative.
The model does not predict a label and it is not the part of the ground truth.
Okay, so if I just show you a true negative over here.
So now you can see that, um, we have our donut over here, and my model does not make any prediction.
Like, for example, over here we need to identify whether they are hot dogs in the image or not.
Okay.
So like there is a donut and my model does not make any prediction over here okay.
So this is the true negative because there is no hot dog inside this image.
There is a donut and my model does not make any prediction.
So the model does not predict the label and is not the part of the ground truth.
So here we are trying to identify whether there are hot dogs inside this image.
And you can see there is no hot dog.
There is a donut.
And my model prediction is not hot dog like my model does not predict and label over here.
So this is true negative.
The third one is false positive.
The model predicted a label, but it is not a part of the ground truth.
So you can see over here false positive.
So here you can see we have an ice cream inside this image by model.
But my model predicted that there is a hot dog.
So there is no there is no hot dog inside this image.
So Ben but there is an ice cream inside this image.
So this is a false positive.
The model predicted a label like no model predicted that there is a hot dog, but it is not the part
of the ground truth.
There is no hot dog inside this image.
That the fourth one is false negative.
The modern does not predict it, but it is a part of the ground truth.
Okay, so now you can see that my model does not predict anything like you can see in this image.
The false negative.
There is a hot dog inside this image like we can see over here, here.
But my model does not predict anything.
Like my model is unable to predict that there is a hot dog.
So the model does not predicted a label, but it is a part of the ground truth, like there is a hot
dog inside this image.
The hot dog is a part of the ground truth by model, but my model is unable to predict anything like
model does not predict that there is a hot dog.
So if you want this both things true positive true negatives, false positives and false negatives,
you will have the confusion matrix.
So the next thing is intersection over union.
So intersection over union indicates the overlap of predicted bounding box coordinates to the ground
truth bounding box coordinates.
So now you can see that here we have a stop sign.
So the green color bounding box represents the ground truth bounding box.
And this maroon color are bounding box represents the predicted bounding box okay.
So.
Using intersection over union, we find the overlap of this predicted bounding box with the ground truth
bounding box.
Okay.
So if I get the higher value of IOU like I do, value ranges from 0 to 1 and I get the IOU value of
0.9.
So higher value indicates that the predicted bounding box coordinates closely resembled with the ground
truth bounding box coordinates.
For example, in this case, I will be getting a higher value because my ground truth predicted bounding
box coordinates, which you can see in the maroon color, closely resembles with my ground truth bounding
box coordinates.
Like, you can see that my predicted bounding box is very close to the ground truth bounding box like
they very closely resemble, like the predicted bounding box.
Very closely resembles with the ground truth bounding box.
So in this case we will have a higher IOU, and we use intersection over union to calculate the overlap
between predicted bounding box coordinates with the ground truth bounding box coordinates.
Okay, so IOU intersection over union metric evaluates the correctness of prediction.
So intersection over union value ranges from 0 to 1.
Okay.
With the help of IOU threshold we can decide whether the prediction is true positive false positive
false negative and true negative.
I told you in the start to calculate true positives false positive we need to calculate intersection
over union.
So intersection over union metrics evaluates the correctness of prediction.
So you can see over here we have set the threshold as 0.5.
And I am getting an IOU value of 0.96.
Like you can see that in the red color I have the ground truth bounding box.
And in the blue color this is my predicted bounding box.
So now we can see that with an IOU is 0.96.
Like you can see that, um, the higher IOU means that the predicted bounding box, which is in the
blue color, closely resembles with the ground truth bounding box.
Okay, so there you can see that we are getting a higher IOU, and you can see that our IOU value is
above the IOU threshold, which is 0.5 over here.
So as our IOU value is above the IOU threshold.
So we can say that this is true positive.
Now you can see that here we have threshold 0.5 and I am getting an IOU value of 0.22.
So now you can see that, uh, my IOU value is less than the IOU threshold which we have defined over
here.
So this is a false positive.
And as I told you at the start, what is false positive.
The model predicted a label, but it is not a part of the ground truth.
So if I just show you like you can see that model predicted hot dog.
But there is an ice cream in that image.
So now you can see that the model makes a prediction over here.
Like but this is not a part of the ground truth.
But this avenue which we got is 0.22.
And it is less than the IOU threshold which is a false positive.
Now the next thing is false negative.
So if I show you over here the model does not predict a label, but it is a part of the ground truth.
Like you can see over here.
I have a bird inside this image.
Like I have the ground truth over here, image over here.
But my model is unable to predict anything.
The model does not predict that there is a label a bird in here.
So in this case, the model predicted that there is a bird.
But over here the model is unable to predict anything.
So this is a false negative.
But like you can see we have a bird inside this image.
But my model is unable to predict that there is a bird.
So this is false negative.
And you can see that IOU is 0.00 which is less than the threshold which we have defined 0.5.
So next is precision and recall.
We will discuss the precision and recall.
So precision measures a precision measures how well you point to positives out of all positive predictions
okay.
So precision is equal to true positives divided by true positive plus false positive.
So uh precision is basically how will we can find true positives out of all total positive predictions.
So for example, if I set the threshold 0.5 and this is my ground truth, uh, bounding box and this
is my predicted bounding box, and I am getting an IOU value of 0.7.
So IOU value 0.7 is greater than the threshold.
So this is a true positive.
And here you can see that I am just getting an IOU value of 0.3 which is less than the threshold which
I have defined 0.5.
So this is the false positive.
So next we have recall.
So recall measures how well we can find true positives out of all ground truth.
So recall is equal to true positives divided by true positives plus false negative.
So recall measures how well it can find true positives out of all ground truths.
Okay.
And the value of recall ranges from 0 to 1.
So next we will see how we can calculate average precision.
So please remember average precision is not the average of precision.
Average precision is basically the area under the precision recall curve.
So here we will be calculating the average precision.
Mainly the average precision is not the average of precision.
Please remember.
Post.
We are using the Yolo V8 model to do object detection on an image like you can see that, uh, we have,
uh, we are using an object model to do object detection on this image.
You can see over here we already have ground truth image.
So this is the ground truth image.
Like you can say this is our input image.
We already have the ground truth image like we have the input image with annotations.
Like you can see that I have uh, this is my ground truth image with annotations.
Or you can say this is my, uh, input image with annotations.
Like you can see that we have annotated person two persons over here.
We have annotated all the dogs.
Over here we have an annotated a teddy bear over here.
So the, uh, my ground truth image has two person one this one one this one and 12 dogs.
So if you count all this, these are 12 dogs.
And I have one teddy like you can see over here.
And this is one truck.
Okay.
So now I will be using Yolo V8 model to do object detection on this image.
And from Yolo V8 model I have these predictions like the Yolo V8 model predicted seven dogs like while
I have 12 dogs in my ground truth image.
But the My Object model predicted that there are seven dogs three Teddy while there is one teddy in
my ground truth image, but my yolo V8 model predicted that there are three.
Riding like this is the teddy, which is a wrong prediction.
This is the teddy, which is a wrong prediction.
One person, although there are two persons, but they detected only one person, one sheep.
There is no sheep.
This is wrong prediction.
And one duck there is one truck.
So these are the predictions from the YOLO model.
So like you can see that we have drop top.
Person, daddy.
So we have multiple classes inside.
Inside this image okay.
Okay.
So we will get creating the average precision class wise.
Like first we'll get average precision for the dog class.
Then we will calculate average precision for the Teddy class.
Then we will get get average precision for the person class.
And then we will calculate the average precision for the truck class okay.
Like you can see over here.
I have just added all the predictions over here.
The light detections like you can see this is the field.
This is false positives.
And here you can see we have true positive true positive false positive.
Like uh this is a false positive prediction as well.
This is not a dog in over here.
So like you can see we have the true positive true positive.
And here we have the confidence score which is provided over here as well.
So these are true positive true positive true positive false positive true positive true positive and
false positive.
So now I will show you how you can calculate precision and recall.
Uh there is mistake like you can see over here.
Uh there we have it.
We should have 12.
Okay.
So we will be dividing it one divided by 12.
Okay, so I told you what is precision?
Bishan is.
How will you find true positive out of total all positive predictions?
Okay.
Or if I show you over here.
So this is I'm using the precision formula of total positives and true positives out of all positive
predictions okay.
And if I show you recall formula over here, recall measures how well you can find true positives out
of all ground truth.
So like you can see that I have 12 ground truth.
Uh, annotations.
I have 12 annotations for the dog class in the input image.
Like you can see we have 12 over here.
Okay.
And recall is how well you can find true positive out of all ground truth okay.
So here you can see this is not 16 in here we should have 12.
Here we will also have 12.
So here we will also have 12.
So we will be dividing it by 12 because we have uh 12 annotations in the ground uh in the input image
for the dog class.
Okay.
So you can see the annotations in the input image for the dog class.
Okay.
So now you can see that in the first prediction this is true positive.
And we have the cumulative true positive one.
And currently we have false positive as zero.
So one divided by one plus zero which you can see our formula over here.
And we have total ground truth for the door class S12.
So if we do divide one divided by 12 this comes to be 0.08.
So you can check it as well.
One divided by 12 is 0.08.
So now next is 0.91.
This is a false positive.
And here you can see we have one.
So now in the formula we get the answer 0.5.
And if I divide one divided by.
612 is 0.08 because in recall we took cumulative true positive.
Now here you can see we have true positive again.
So it comes to and last one.
So you can see over here.
Two divided by five.
It turns out to be 0.66.
Biologist.
There is a mistake as well.
So here we should have.
One over here as well.
So I just add this over here.
Okay.
So.
So now you can see if you divide two divided by three it comes out to be 0.66.
Because here we shouldn't have three because two plus one.
Okay.
And here we have two.
And if you divide two by wealth you will get 0.16 in output okay.
Then you can see we have false positive.
And in the similar way you will make this calculations as well.
And we will make this calculation for all the predictions that you get from the V8 model.
And now we will plot the CNN graph like you can see over here.
Uh, so here we will be using uh, calculating average precision using Pascal VOC 11 point interpolation
method.
So to calculate average precision I will be using Pascal VOC 11 point interpolation method.
The Pascal VOC 11 point interpolation method was introduced in 2007.
Pascal VOC challenge, where uh precision values are recorded over 11, equally spaced.
Recall values.
Average precision is defined as follows.
Average precision.
Average precision is one divided by 11 sum of 11 interpreted precision.
So in the x axis you will have the.
Recall, and in the y axis you will have the precision.
So precision values are interpolated across 11 recall values.
The interpolated precision is the maximum precision corresponding the recall value greater than the
brand recall value.
So you can see that, uh, from the values which I have bought over here.
Okay.
So I have been using these values and I have created this precision recall graph.
Okay.
So basically I told you at the start average precision is the area under the precision recall curve
okay.
So like you can see that I have kept lot final interpolated graph and calculate average precision for
that dog class.
So you can see that I have created this precision recall curve over here.
And you can I show I told you at here as well.
The interpolated precision is the maximum precision corresponding to the recall value.
So if I just see over here this is the maximum precision.
Over here I can see this maximum precision value over here.
So.
The interpolated precision is the maximum precision corresponding to the recall value, so this is the
maximum precision.
And here you can see I've created the average precision for the dog class.
So if you can see over here at 0.0 we have 1.0.
So we have only one uh for was the at 0.0 and 1.054, 0.1, 0.2, 0.30.4, I am at 0.71.
So for multiply by 0.71.
And in the next six values like 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, I am at zero.
So six multiply zero.
And I am just getting average precision for the dog class as 35 4.9%.
So in the same way, I will get average precision for the person class and just calculate this manually
on paper.
So I as a same approach, I'm just calculating the precision and recall values which I just followed
over here.
So and I'm just completing drawing the precision recall curve.
And you can see over here I'm just getting a value of 0.545.
And in the same way I'm also calculating the average precision for the drug class.
And here I'm just getting the average precision value as one.
The same approach I'm following.
Just I told you about.
And here I'm just calculating the average precision of the sheep and teddy class.
And the same approach which I showed you about.
And here I am.
This mean average precision.
So mean average precision is the average of average precision over all detected classes.
So I have you can see that if I just show above over here.
Okay, so my model detected dog teddy person sheep truck.
So five different classes that my model detected.
Okay.
And.
I just calculated the average precision for each of these class as well.
Like I showed you over here, I could put average precision for each of these class.
So we have total number of five classes.
And I just kept pretty average precision for each of the class which I showed you above.
Okay.
And I'm just getting a mean average precision score 47.88%.
So here you can see that, uh, I am what I am using is I am using Pascal VOC level point interpolation
method.
So I have used Pascal VOC 11 pointer interpolation method to calculate average precision okay.
So calculate average precision using Pascal VOC 11 point interpolation method.
Okay.
So.
So here you can see in Pascal VOC 11 by integration method I average precision is calculated at IOU
threshold of 0.5.
Okay.
But currently, uh, the 101. iteration method is used to get rid average precision.
So currently in object detection models like YOLO, YOLO, yolo v7, Yolo v6 101 Inter point interpolation
method is used to calculate average precision, so Mscoco introduced 101. interpolation method to calculate
average precision in 2014.
It is better approximation of accuracy under the precision recall curves okay.
In Pascal VOC 11 point term regression method.
Average precision is calculated at IOU threshold of 0.5, like I showed you above.
So we have calculated the average precision considering an IOU threshold of 0.5.
But currently the approach which is followed is mscoco 101.
Interpret 101. interpolation average precision.
So in this case uh, the average precision uh, or you can say the mean average precision is being calculated,
uh, considering a set of ten different thresholds, which range from 0.5 A to 0.95.
So in current mscoco 101 interpolation method, the mean average precision is calculated, uh, from
IOU 0.5 to 0.95 and with a step size of 0.05.
So coco mean average calculate mean average precision is calculated for a set of ten different thresholds
and then average.
It ranges from 0.5 to 0.95 at a step of 0.05.
So before I miss Coco, the mean average precision was calibrated at higher threshold of 0.5.
But now the mean average precision is calibrated at an I o threshold from 0.5 to 0.95, with a step
size or step frequency of 0.05.
At present, Miss Coco introduced 1.1101. interpolation.
Average precision is accepted is accepted at the standard metric.
So currently Miss Coco introduced 101. interpolation.
Average precision is being accepted as the standard metric, but in this tutorial we use Pascal VOC
11 point interpolation method to calculate average precision.
So that's all from this tutorial.
Thank you for watching.