Hi and welcome to our lecture on mean average precision.
This is one of the most important metrics that we use to evaluate object detector model performance.
So let's get started.
So firstly, in summary, mean average precision is perhaps the best metric to look at when comparing
models on the same dataset.
That's why it's it's become the metric that we use in competition, Steve.
Other medium models are competing against each other, and it's actually the aggregation of several
metrics which we'll talk about now.
So firstly, just to recap or record no pun intended.
Let's take a look at what precision and recall were.
So precision is essentially when your model predicts positive.
How often is it right?
So what does that mean?
That means, let's say we have a model that's detecting dogs, and every time it predicts a dog, how
often is that right?
That's basically the precision, the accuracy of that, because sometimes it can be predicting a horse
or a cat or dogs.
And therefore, it's wrong.
So that's why it's the true positives over the true positives plus two false positives.
OK, now what about recall?
Recall is another very important metric.
It basically tells us how well our model is performing at finding all the classes it says it's supposed
to find.
So finding all the positives essentially in a more simplified way of thinking about it.
So what does that mean?
That essentially means if we have an object detector that is looking for dogs, we want to know how
much times of missed dogs overall so that we can develop a score to know how well it's recalling dogs
in that dataset.
So how well a model is at finding dogs, essentially.
So now let's take a look at how we use precision and recall in the object detector world.
So as I said with this example, let's say we have an object detector that is detecting dogs here,
and our test dataset is comprised of just these tree images.
You can see here that this is clearly not a dog.
This is a cat.
So what does that mean?
It means our model predicts a dogs three times, but only two times the model is correct.
So that means our precision is to of a tree or two thirds or sixty six point six percent.
Not that great, but it depends on, you know, how hard your task is.
Now let's take a look at recall, and the object that I talked to will recall is how good of a model
is at finding for detecting an object.
So namely, we have this new test dataset with these three images here, and we can see that it missed
this dog, unfortunately.
So that means that we we predicted two dogs, but missed one out of the tree.
So I will recall again, it's not two of a trio of 66 percent.
So let's think about something, though what if we changed our I.O.U. threshold?
Remember what are you?
Threshold tells us basically how much overlap in the boxes we we we use.
So in this case, this is like the minimum of overlap that it will be.
This is 50 percent.
Let's see.
So it's overlapping with 50 percent here.
That's the union.
And what's happening now is that if this box were a little bit bigger, it would drop out entirely.
There would be no bounding box at that point because it's been lowered a threshold.
So as you can see, if we changed our view threshold, let's say we widen.
So let's say this box was a bit wider than it actually should have been.
So we allow more flexibility and more leeway with our model by reducing the IAU threshold.
We can now realize that by making it less strict means no that we can get different precision and recall
values.
So you can see the stricter we make you threshold.
Like, let's say it was 90 percent.
It wouldn't show this monkey box anymore, but it still would show these two.
So that's basically how the I.O.U. thresholds affect our precision and recall.
No, let's think or talk about map.
So Map stands for mean average precision.
So firstly, average precision is basically finding the area of a under precision recall curve with
varying you confidence thresholds.
So you can see an example here.
Let's think about one class as we could the coffee more class as the pink line here.
So you can see this is the precision here, how much opposition is.
And this is a wrinkle here, and you can see if we set an IQ threshold of, let's say, zero at that
point.
No, this is this access is not you.
By the way, this access is just precision.
And recall this, for instance, I'm just saying if we set an IQ threshold of zero is probably not a
good one because you're going to get all boxes at that point.
But that means that you'll get high precision score and very low record score.
That's why it's.
Zero.
So that's what's happening at that point, or it could be the reverse.
You have to think about it a bit.
But either way, what's happening, though this graph is that we're actually plotting the recall and
precision for all the different EU thresholds from zero to one.
And we get this curve here.
So this curve is called an average precision curve.
So what is the mean average precision?
Well, the mean average position is basically this blue line.
Now it's all of them averaged out as well here, so you can see the formula.
So we just get the average position for each class and we just basically find it to mean average precision
out of it.
It's essentially that simple, although this is a very confusing metric that a lot of people don't fully
understand.
So it's important that you do understand what's happening here and why is this such a good, good metric
or measure to evaluate this?
Well, you can see it considers a number of things it considers.
Firstly, the classifier accuracy how good we are detecting objects, the correct objects, being at
the correct class, you know, object detection model, and it also considers the bounding box locations.
So it's an aggregation of different metrics that evaluate classification and the bounding box accuracy.
So that's why it works so well as a benchmark tool metric for other sectors.
And it's what's used commonly in the KushCo dataset that sense for the common objects and context dataset.
This is basically the image in that for the object detection world.
It's a very vast dataset that actually it's constantly improving, but we have different visions that
we test on for the competitions.
And you can see right now it's as image screenshot captured from pictures from people with code sorry,
and you can see these are the best performance right now using different APIs or APIs.
Basically, the sense a map.
Essentially, it's not just the average precision, average precision for world classes, and you can
see they have different metrics as it is 50 75 AP Small, medium, large.
What did they use?
Extra training data so you can see these are these are essentially the best performing up to detecting
networks right now in research.
However, I would see a lot of these networks onto practical, and that's why YOLO, which is a very
easy to train, practical, easy to deploy model, has taken over the industry by storm for the last
three or four years, or pretty much all objects that the models used in industrial YOLO, although
efficient, detect and detection to gaining in popularity as well.
So that's it for this lecture on lap.
What we'll do next, we'll take a look at non maximum suppression, which is a technique we use to clean
up our bounding box proposals in optic tech, the detectors.
So I'll see you there shortly.
Thank you for watching.