In this video tutorial, we will learn how we can fine tune the YOLO 11 instance segmentation model
on a custom data set.
We will find you on the YOLO 11 instance segmentation model on for potholes detection.
And here is a quick demo.
So you can see that we are able to detect potholes in this video.
And you can see that our results look quite good.
So we will be fine tuning the YOLO 11 instance segmentation model on potholes dataset.
And we will be detecting the potholes in images videos.
And you can also detect the potholes on the live feed as well.
And uh, we will be training the uh or fine tuning the YOLO 11 segmentation model on potholes data set,
which is available on Kaggle.
So let me first show you the data set.
And so here you can see we are at Kaggle.
So here you can see we have the pothole image data set available over here okay.
so you can see we have around 600 images over here.
And you can see we have different potholes.
And all these images are annotated as well.
So we have the annotation results in a dot yml file over here.
Okay.
So you can see over here um that here we have the data set.
So uh we just first need to download this data set.
So you just need to sign in if you have already have an account on Kaggle.
But if you don't have an account on Kaggle you can, uh, create your own account on Kaggle.
It's very simple.
So you can see I've just, uh, log in with my own account on Kaggle and you can just click on download
over here and you will be able to download this data set.
So it will be downloaded in zip format.
Okay.
So I've already downloaded this data set.
So I will not download it again.
After this I will just go to Roboflow.
Uh, we are using Roboflow so that we can convert our data set, um, into a format so that we can,
uh, train our YOLO 1110 segmentation model.
Okay.
So I just need to sign in.
But if you have done creating your account you can just create an account on Roboflow as well.
So this will take a few seconds and I will assign Google.
But if you haven't created account you can just create an account with your GitHub account with your
work email and use what ever you want.
So there are multiple ways through which you can create an account on Roboflow.
Okay, so here you can see all your projects and you can see I've just created this project six days
ago as well.
But I will not be using this.
I will just create a new project so I can show you how you can, uh, create, convert your data set
into a format so that you can fine tune the YOLO 11 instance segmentation model.
So what Wolds texture And segment pitch.
So this is the name of my project.
Pothole selection segmentation.
And we are detecting the pothole.
And you can see we have the instance segmentation.
Okay.
So detect multiple objects and their actual shape.
Best for measurement and pot shapes.
Okay.
So let's get started.
Create a public project from here.
Okay.
So now you can see we have instance segmentation project not object detection.
Then we need to upload data.
So I will just click on select folder from here.
Okay.
So um you can see over here I have this folder.
And I will just upload this folder complete.
Because inside this folder I have the train validation folders.
So I will just upload this complete folder over here.
And you can see we have around 1564 files.
So we have around 750 images um 760 images around folder.
are images and we have.
So basically our data set is integrated.
So our half files are called annotations.
And half files are basically images.
Okay.
So you can just upload this complete data set over here.
And this will take uh not very much time.
Like okay.
So there is a video as well uh in this folder.
So we need to extract one frame per second okay.
That's fine.
So all this data set will be uploaded over here.
So this will take some time.
And let's wait and see as this data set gets uploaded over here.
So this the files are uploading.
You can see the files are being processed over here.
Okay.
So let's see.
Hopefully this will finish very soon.
Okay.
So now over here you can see that, uh, we will be using 70% of our data for the training, 20% of
the data for the validation purpose, and 10% of the data for the test purpose.
Okay.
So we have around 796 images, and out of 796 images, 780 images are annotated and 16 images are not
annotated.
So I will not try to annotate like we have only one class as we are taking part.
So 780 images are enough.
So I'm not uploading these files in my project which I have created like potholes prediction.
So this will take few more seconds over here.
Okay, so let's see how does it goes.
So now you can see that, the files are being uploaded over here.
Okay, so now you can see in my data set I have 780 images.
Um, I don't want to rotate my images.
Just close this.
Okay, so now you can see over here our images are being generated and you can see over here we have
detected pothole.
Pothole.
Okay.
We have detected pothole.
Pothole.
And you can see that the name of the class is very large.
So we can just update this as well.
So you can see that all the images over here are updated.
Okay.
And we can just check this so you can see that the annotations are quite good.
Like you can see over here.
All the images are annotated.
Okay.
So next thing we can just go to generate over here.
Okay.
So we have one class and we have separate images.
Okay.
So we are taking 70% of the images like 544 images for the training, 156 images for the validation
and 10% images like 80 images for the test purpose.
Okay.
So now we will do some pre-processing.
We will resize each image to six 4640.
Okay.
So one thing we can say is we can add further pre-processing step as well.
Or we can just modify the class name.
Like we can just update this class name to point four okay.
Okay.
There are other pre-processing steps as well.
So if you don't have enough data for the training, uh, you can just increase the size of your training
data by converting your images to grayscale.
Okay.
And, uh, like following other pre-processing steps, like crop some images, uh, or just isolate
objects.
So they are multiple pre-processing steps that you can follow if you don't have enough data for the
training purpose.
Okay.
And where we are resizing our image to size six 4640, because our YOLO 11 model has been trained on
an image size of 640 cross 640.
So it's always better that you fine tune.
When you fine tune your model.
You just, uh, have the image of size 640.
640 because if you increase the size or decrease the size, um, that might affect your results.
Okay, so and if you don't have enough training data, then you can increase the size of the training
data by, uh, converting these images to rotation and rotating.
These images are cropping these images and rescaling these images, adding, uh, increasing the brightness
of these images.
So they are multiple augmentation steps over here that you can follow.
Okay.
Now I will just, uh, generate the data set over here, and then I will export this data set from Roboflow
into my Google Colab notebook.
So now you can see over here my data set is ready.
So now I will just download this data set.
So I just want to download this data set in your 11th format okay.
And um it will show me download code over from here.
So I will just copy this code from here okay.
And I will just go over here okay.
So this is a Google Colab notebook which I have created for this project.
And I have written all the code.
I will just show you a step by step guide how you can fine tune the YOLO 11 instance segmentation model
or potholes detection.
And you can just remove this and you can just add this, uh, code over here.
So now if you run this cell, you will able to download a data set from Roboflow into your Google Colab
notebook.
Okay, so first of all, before running the script, please make sure that you have selected the runtime
as key for GPU.
And then, um, we will just run these cells.
Okay.
So first we will install the Ultralytics package and OpenCV Python package.
Uh so we are using uh YOLO 11 model and YOLO 11 model is available under Ultralytics package okay.
So therefore we are installing the Ultralytics package.
So you can just simply run YOLO 11 model by installing installing the latest package.
You don't need to clone the GitHub repository, although you can clone the GitHub repository and you
can just follow this lengthy procedure as well.
But this is a very simple procedure.
Like you just install the analytics package and simply, uh, run the YOLO 11 model for command line
interface or using Python script.
Okay.
So now from Ultralytics we will import YOLO.
And if we import YOLO Then we can either use that or YOLO model, YOLO 11 model, or any other model
as well of YOLO series.
Okay.
And then we have from IPython dot display import image.
If you want to display an image or in Google Colab notebook we need the image library.
Okay, so now we are downloading the data set from Roboflow into the Google Colab notebook.
Okay, so here you can see this is the API key.
You can just create your account on Roboflow and get your own API key as well.
Please don't use my API key.
Try to get your own API key as well.
Okay, so now you can see the data set is downloaded and you can see we have provided our data set over
here.
Okay.
So now over here you can see we have our data set.
And if you just open the data dot yml file over here.
It's taking so much time.
That means let's save this first.
And this shouldn't take so much time.
Let's do it.
Okay, so here you can see we have the dot yml file over here.
So you can.
First thing you can do is you can just update the path of the train images.
Validation images.
And test it if you just skip this test with this will not this.
This works as well because during training we will be using the training images.
And.
For the validation we will be using the validation data set images okay.
Okay, so please make sure to update the data dot yml file paths for the training in validation and
for the test as well.
Okay.
And you can see we have only one class which is pothole.
Okay.
So everything looks perfect.
Okay.
So now over here we have the dataset location over here as well.
Okay.
And one thing we can do over here is we can just copy this from here.
And we can just add this path over here.
Okay.
So you can download the YOLO 11 segmentation model from uh, the YOLO 11 GitHub repository will be directly
downloaded over here.
So if you write model is equal to YOLO YOLO 11 s dash segmentation.
And this will download the YOLO 11 segmentation model.
YOLO 11 comes with five different models YOLO 11 nano, YOLO 11 small, YOLO 11 medium, YOLO 11 large
and 11 Extra large.
So we will be using YOLO 11 small model.
If you want to get more accurate results, you can use the higher YOLO models.
But um, so, uh, what, uh, what is the difference between your 11 small models and your 11 large
models?
So a YOLO.
11 small models are faster, but they are less accurate.
But YOLO 11 large models are more accurate, but they are less, uh, fast or they have, uh, don't
have, uh, they are not very fast or the inference speed is slow in, uh, for the YOLO 11 large models.
Okay.
So basically we are trying to perform instance segmentation.
And first we will train the YOLO 11 uh segmentation model on for pothole detection.
And here you can see we have the data dot yml file path over here in the data dot YAML file path.
Uh we have a train validation test images path over here okay.
And we are trying to fine tune the YOLO 11 instance segmentation model for 50 epochs which you can find
over here.
Okay.
So uh, we are not changing any of the parameters.
Like you can see over here.
Our patience is being set to 100.
And uh, our batch size is being stretched to 16.
Okay, so what does batch size mean?
So we will be passing our data to the model in 16 batches okay.
I will later show you how this what this means as well.
Okay.
So here you can see I've already fine tuned the YOLO 11 instance segmentation model for pothole detection.
And you can see I have just fine tuned it for 50 epochs.
I will not run the training again because this takes a lot of time as well.
And you can see over here we are getting a mean average precision with IOU 0.5 or 50% is 0.908 or 90.8%.
And mean average precision with IOU varies from 0.5 to 0.95, is 0.623 like 62.3%.
And these are the bounding box prediction results.
And here we have the mass prediction results over here.
And they are quite good as well okay.
Okay.
So over here if you just see over here our batch size was set to 16.
As I told you.
Okay, so, um, if I just show you one thing.
Um.
So I told you that we will be passing our data in 16 batches.
Okay.
So over here.
So as we will be passing our data in 16 batches, uh, here you can see over here, uh, and our default
image size is six 4640.
We are not changing this.
So if you just see over here, we have 543 images in our training data.
And you can see 34 divided by 34.
So what does this mean.
So if I just open my calculator over here okay.
So we will be passing our data in 16 batches.
So if I just divide 543 by 60.
So we have 33.93 okay.
So in each batch we will be passing 34 images okay.
So our complete data will be passed in 16 batches, and in each batch we will be passing around 34 images.
Okay, so this is what it means over here.
Okay.
And, um, our results are quite good.
Like, uh, for the bounding box predictions, we are getting a mean average precision with IOU threshold
as 0.5 or 50% as 90.8%.
And for the mass prediction, we are getting a mean average precision with IOU threshold 0.5 or 50%
or 0.91 or 91%.
Good.
And after I have, as I told you, that I have already trained the YOLO 11 instance segmentation model
for on potholes uh data set.
So I've just saved the model weights on my drive.
So I will directly download the weights from my drive into this Google Colab notebook over here okay.
Okay.
So over here now you can see that, um, these are some results over here.
So now you can see over here as we increase the present value of our.
Now you can see over here on the x axis we have the confidence value.
And on the y axis we have the precision score.
So you can see as the confidence increases um our precision value increases.
Okay.
So what does precision mean.
Precision is basically a sum of true positives or true positives divided by all the positive predictions.
So if I just you just write precision in computer vision.
So you can see over here.
The okay as precision medicine.
How will you find true positives out of all positive predictions okay.
Okay.
So here you can see we have the general formula.
Uh okay.
Over here.
So you can see that, uh, true positives and I just based.
On no es un.
Okay.
So here you can see that, uh, a correct predictions divided by total predictions.
Okay.
So this is precision correct predictions divided by total predictions okay.
So over here, for example, uh, you made five predictions and four among you have ten ground truth.
And I made five predictions for that.
Two positives.
And there is one false positives okay.
And there are six false negatives like we are unable to detect uh, ground truth six times okay.
So like precision will be 80%.
And uh recall we will discuss later okay.
So basically we uh check how many correct predictions are made out of total predictions.
Okay.
So as you can see that as we increase the confidence score okay.
So when we increase the confidence score, our predictions might decrease.
Okay.
But when we increase the confidence score our model, make correct predictions like when.
If we can say that if our detection has a confidence score of 0.90, this means that the model is 90%
confident, uh, that, uh, this is a box or this is a pothole.
Okay, so as we increase the continuous threshold, our precision score will definitely increase.
Okay.
Because when we increase the confidence score, we will be getting, uh, more correct predictions,
and, uh, there will be less false positives.
Okay.
So over here and we can see that when, uh, we have a confidence score of 0.946, then we have a precision
score of 1.00.
Okay.
And over here you can see that as we increase the confidence for our recall value point begins to decrease.
Okay.
And, uh, and you can see that, uh.
And then we have a confidence score like 0.0.
Then we have a maximum recall.
So if you just see over here.
Recall is basically correct predictions out of total ground truth.
So you can see over here if we have ten ground truth uh, images of dogs or ten ground truth of dogs,
and we make ten predictions out of this, we have eight true positives and two false positives and two
false negatives.
So we have like, uh, 80% of recall.
And you can you can see that if you make five predictions like five, we have ten ground truth for the
gag and we make five predictions.
And out of five predictions we have, uh, four true positives and we have six false negative.
And in that case we might have a higher precision, but the recall is 40%.
Okay.
So when we increase the confidence score like you can see, you know that our predictions will definitely
decrease.
Okay.
So When we increase the confidence score.
Our predictions will decrease.
And you can see that, uh.
Uh, I will be not be able to detect all the ground truths.
So therefore the recall value will decrease.
So when we have, uh, a confidence score of 0.00, then we have, um, maximum recall and confidence
score 0.00, which is 0.96.
So we have a maximum value of recall 0.96 at confidence zero zero.
So these are predictions for the bounding boxes.
And as instance segmentation we not only predict the bounding boxes we also predict the mask as well.
So these are the graph plots for the mask.
And you can see they are quite similar with the bounding boxes as well.
So over here we have the confusion matrix.
So from confusion matrix tells us how the model handles different classes.
So um for 29 times when there is a pothole or our model predicted correctly that there is a pothole.
And 51% of times when there is a pothole, our model is unable to detect that there is a pothole.
Okay.
And 89% of the time, our model detected correctly that there is a pothole, and 11% of the times our
model is unable to detect that there is a pothole.
So these are the model predictions on the validation batch.
We are not using the validation images data set images for the training.
So it's always better to have a look and see how our model performs on the validation dataset images.
So you can see that results look quite promising.
And over here you can see that, um, our accuracy tends to increase or the mean average precision tends
to increase as being as the number of epochs continue to increase.
Okay.
And now we will validate the fine tuned model.
So to see um, uh how our model generalizes.
So we are validating our model on the validation data set images to see how our model generalizes.
And you can see that um for the box predictions we are getting a mean average precision with IOU 0.50.909
1909 like 19.9%.
And for the mass predictions, we are getting a mean average precision with IOU 0.50.91 like 91%.
Okay, so now we will do perform inference on the test dataset images over here.
Okay.
So what you can do over here is I'm just checking one thing.
Okay.
That's fine.
So you can just go over here and just copy this part from here.
And you can just add this part over here.
And you can just perform inference on the test dataset images as well over here.
Okay.
And you can just display this results over here as well.
I've just written some code using matplotlib library over here.
Okay.
So it will display all the model predictions on the test dataset images over here.
So now this will be.
Take your seconds.
Okay.
So here you can see we have the model predictions on the test data.
Set images.
And we will see over here as well.
So this will take few more seconds.
And let's see what results do we get over here.
After this we will test our model on this two videos.
Okay.
So uh let's see I think this shouldn't take as much long, but let's wait.
So these are the model predictions on the test dataset images.
And you can see that the results look quite good.
We are able to detect the potholes everywhere.
Okay.
So that doesn't look quite good.
Yeah.
But here you can see there are many potholes we are not able to detect all.
So if you just, uh, train your model on high number of epochs and you will be able to detect these
as well.
Okay.
Or you can just, uh, decrease the confidence score over here and test this as well.
You might get more accurate results.
Okay.
So let's test our model on this sample video over here.
Okay.
So you can just run this as well.
And you will able to, uh, get the potholes in this video.
So now you can see that, uh, the complete video is being divided into 324 frames, and we are detecting
potholes in each of the frame one by one.
So now I will display this video into the Google Colab notebook over here.
And then I will just show you the results as well over here.
Well here is our output video.
I will just simply download this video from here.
And.
here you can see is our output.
So you can see over here we are able to detect the potholes.
And our results look quite promising.
Like you can see we are able to detect the potholes over here in the same way you can test on other
video as well.
I've already done the testing and here I have the output video.
So I will just simply download this output video and show you, uh, what results look like.
So over here you can see we are able to detect the potholes as well.
So in this tutorial we have seen that how we can fine tune the YOLO 11 instance segmentation model for
potholes detection.
You can also fine tune the YOLO 11 instance segmentation model for to detect any other uh, object as
well.
So you can just fine tune the YOLO 11 instance segmentation model or any other custom data set as well.
So that's all from this tutorial.
Thank you for watching.