In this video tutorial, we will look at how we can fine tune the YOLO 11 code estimation model for
human activity recognition.
And here is a quick demo.
You can see that we are able to detect a fall.
Along with this, we will also be able to detect other body positions like sitting, standing, walking
and fall down as well.
So let me first show you the data set.
And over here you can see that we will be using Human Activity Keypoints data set available on Roboflow.
And we have total 1608 images in our data set.
And you can see over here we have a fall down sitting standing walking.
So we have these different classes over here.
And we have two versions of data set available.
And like this one.
And this one we will be using the second version.
In the second version no augmentation is applied.
So that works because we have enough number of images in that training data.
So we have 70% of images like 126 images in the training data, 20% of images like 324 images in the
validation data.
And we have 158 images in the test data set, which account for 10% of our total data set.
And um, we have I think we also have fine tuned models available over here as well.
And um, our analytics like just check the model health check up for class balance.
You can see that we have good number of images for the sitting class fall down and standing.
But the walking class is underrepresented.
Like this.
The data set is not totally balanced like walking class is underrepresented.
So if you are in ideal scenario, we should add more images into our walking class for the walking class
and try to make it like increase the number of images or Our agitations for the working class and try
to make it equal to the are at least standing class as well.
Okay, so like in all the three scenarios, training, validation and test are the working classes are
underrepresented like the dataset is not completely balanced.
Okay.
And here are some insights like where our majority of annotations lie okay.
And our image ratio is 1287 20 okay.
So these are different images we have.
You can see over here.
So now you can see that uh the images are properly annotated as well.
So here we have the human activity Keypoints dataset that is available on Roboflow.
In order to use this dataset in your, uh, project or in Google Colab notebook, you need to create
an account on Roboflow and then you will be able to download this data set.
Or you can export this dataset from Roboflow into the Google Colab notebook, so you can just select
the format from here, like it show download code, and you can just copy this code from here and add
it to your Google Colab notebook.
And you will be able to download this data set from Roboflow into the Google Colab notebook.
Or else.
Uh, there is another way that you can download the zip file to your computer by clicking on continue.
This will start downloading this zip file to your computer, but I have already have the zip file so
I don't need this.
I will cancel it.
Okay, so now here I have just created this complete notebook and it's completely ready.
So before running the script, please make sure that you have selected the runtime SD or GPU.
So in the step number one will install that politics package.
So now you can see the Ultralytics package is installed.
So now we will import Ultralytics.
And from Ultralytics we will import YOLO so that we can access the YOLO 11 model.
So now we are checking the analytics like which version is available and the Python version as well,
and how many whether we are using GPU or not.
So we have our latest latest 8.3.12 version available.
We are using Python 3.10 and we have Cuda available, Tesla T4 GPU with 15 GB Ram, and we have two
CPUs available, and currently we have a total disk space of 112 GB, out of which 36.6 GB is being
occupied.
So I try to download the data set from Roboflow into the Google Colab notebook using this piece of code,
like when you just click on Download Data set and you select the format, um, and using this, copy
this code and add it over here.
So but it gives me the error I can just show you as well.
So.
So if you try to download that data set using this piece of code, it will give you an error.
Okay, so previously it was giving me an error, but now it is working fine.
So you can use this way as well to download the data set.
Now you can see we have downloaded the data set from Roboflow into our Google Colab notebook.
So that works fine are pretty fine.
Like I'm just amazed.
So we can just or you can just.
I have also downloaded this data set into the zip format like I was trying, and I have just placed
this data set into my Google Drive.
So now I'm directly downloading this data set from my drive over here.
Okay.
So either you can um, just download this data set and just create another folder in the name of Gregory.
And you can just unzip this data set over here.
So you can see we have a data set folder and you can see where we have the data dot yml files over here.
Okay.
Let me just save this script for now.
So there are two ways like you can follow either of the way okay.
So either you can just download this data set from drive into your Google Colab notebook.
And uh, just save this unzip this data set and you can see we have this data set.
And uh, you can see here we have the downloaded the data set from Roboflow into the Google Colab notebook.
So if you just follow this, uh, if you just export the data set from Roboflow into the Google Colab
notebook, then you need to update the data dot yml file path over here.
Like you can see we have the names.
And if I just check the data dot YAML file, this one as well.
So you can see we have the same things available over here as well.
So.
So it's your choice.
You can use either of the way approaches.
Okay.
But one thing you need to make sure is if you want to use this data set which you downloaded from Roboflow
into your Google Colab notebook.
So you just need to update the training and validation path.
So you can just go over here and you can just copy the path over here.
We make sure to update this path to paths over here.
Just copy path from here.
Okay.
So please make sure that you update that training and validation paths over here.
After this you can download the YOLO 11 uh for the estimation model.
So I'm directly downloading YOLO 11 pose estimation model from GitHub repository into this Google Colab
notebook.
And then I'm also loading the YOLO 11 model, and the YOLO 11 model is saved in the variable model.
And I'm using YOLO 11 small pose estimation.
Currently YOLO 11 comes with five different models YOLO 11, nano small, medium large extra large.
So the difference between each of these models, like YOLO 11 nano is the fastest, but it is least
accurate.
But YOLO 11 Extra Large is the most accurate, but among other YOLO 11 models.
But it is takes more inference time.
Okay, so we are using currently using YOLO 11 small for the estimation model over here.
And I have downloaded the model and I have saved the model into the variable model.
Okay.
Then I will not run the training again because I have already done the training and I was not recording
this video, I was just preparing this notebook.
Then I just run the training.
So now you train the YOLO 11 or fine tuning the if you want to fine tune the YOLO 11 or estimation model
on human Activity data set.
You will do model dot train.
Here you will pass the data dot YAML file path and we are finding the YOLO 11 code estimation model
mode is equal to train, and we are fine tuning it for 50 epochs.
And the batch size is being set to eight.
So currently you can see over here we have 1126 images uh in our training data.
So these images will be passed in in eight batches okay.
So here you can see.
So if you just open the calculator from here and.
And if you write 1126 divided by eight so you can see 140 .75 like it's 141.
So we will be passing our data in eight batches.
And in each batch we will pass 141 images as we have 1126 images in total.
Okay.
So here you can see that, uh, these are the mean average precision.
Like in this case, we are doing two things.
First, we are predicting bounding boxes.
Plus also we are doing the prediction or estimation of key points.
Okay.
So this is the mean average precision with IOU 0.5 for the bounding boxes.
And these are the mean average precision with IOU 0.5 for the key points prediction okay.
And uh it will run for 60 epochs.
And you can see over here, as I told you, that we have around four different classes in our data set.
And over here you can see that, uh, we have the results.
Like for the working class, I told you that the data set contain less number of images, so we cannot
expect the same performance, uh, for the working class as compared to other classes.
So now you can see over here we have the fall down class sitting class standing walking.
And you can see over here, um, even for the walking class, we are getting a good, mean, average
precision score.
And, uh, in for the pose estimation prediction.
Like for the working class, you can see that, um, accuracy is low.
Like you can see that for all the other classes the accuracy is above 90%.
But for the working class the accuracy is less so in terms of pose estimation prediction.
For the working class, the accuracy has compromised.
Okay.
So then we are validating our model.
Uh, so on the validation data set.
So we basically validate our model uh to see how our model generalizes.
So now you can see that we are good getting good mean average precision score over here okay.
So like the results look promising for me okay.
So now here we have the confusion matrix.
So um 88% of the time when the person has fallen down, our model predicted correctly that the person
has fallen down.
While 2% of the times when the person has fallen down, our or model predicted that a person is sitting
while a one person or 1 to 88.
One time when the person has fallen down.
Our model was unable to detect anything.
Okay, so I have, uh, while explaining I made might made a mistake.
So this is not 88%.
So this is 88.
Like what, 88 instances?
Okay.
So in for 88 instances when the person has fallen down, our model predicted correctly that the person
has fallen down and 2% are two times when the person has fallen down.
Our model predicted that the person is sitting while one time when the person has fallen down, our
model was unable to predict anything and 1104 times when the person is sitting.
Our model predicted correctly that he is sitting while 74 times when the person, uh is standing.
Our model predicted correctly that the person is standing while nine.
Times when the person is standing.
Our model predicted that the person is walking, while one time when the person is standing, our model
predicted nothing.
Okay, while for the 41 times when the person is walking, our model predicted correctly that the person
is walking, while five times when the person is walking, our model predicted that the person is standing.
Okay, so in this way you can, uh, use the confusion matrix.
So confusion matrix tells us how our model handles different classes.
So this is a normalized to 97%.
So here is the percentage.
So 97% times when the model when the person has fallen down.
Our model predicted that the person has fallen down.
While 2% of the times when the person has fallen down.
Our model predicted that the person is sitting while 100% of the times when the person is sitting our
model predict correctly that the person is sitting, while 888% of the times when the person is standing,
our model predicted correctly that the person is standing, while 11% of the times when the person is
standing, our model predicted that the person is walking, while 1% of the time when the person is
standing, our model is unable to predict anything.
While 87% of the time when the person is walking our model predicted correctly that the person is walking,
while 11% of the time when the person is walking, our model predicted that the person is standing.
So in this way you can use the confusion matrix.
So here comes the precision curve.
Um, before we look at this curve, let's understand, uh, what is precision.
Okay, so if I just said precision computer vision.
So if I just go over here and here.
Okay.
So over here you can see that precision is a total correct predictions divided by total predictions.
Okay.
Uh, so basically you can see that um, precision is basically total true positives divided by total
positive predictions.
Okay.
So like um here we have the formula for precision.
So if you just see this curve as we increase the confidence threshold the precision will increase.
Because when we increase the confidence threshold our false positives will decrease.
Like we will be getting less false positives.
So when we increase the confidence, we will definitely getting less false positives and we will getting
more true positives.
Okay, so over here and when we get more true positives, we will have definitely have a more better
value of precision.
Okay.
So over here you can see that when we have a low confidence threshold we might be getting a very high
false positives.
So we will have low uh precision value at low confidence score.
As we increase the confidence threshold our precision value will increase because, uh, when we increase
the confidence or our false positives will decrease, our true positives will be increasing.
Okay.
So therefore, when we increase the confidence threshold, the precision value will increase.
And you can see over here, uh, for all classes the precision value is 1.00.
Like it's the perfect one value at 0.975 quantile.
So at 0.975 confidence threshold, we are getting a precision value of one for all classes like you
can see over here as well.
While you can see for the working class as we have less number of images for the working class, I showed
you in the training data that it's under balanced or underrepresented.
So you can see the precision score is low for the working class even at higher confidence threshold,
even as you can see higher confidence threshold and the precision score is low for the working class,
while for all the other classes we are getting a very good precision score.
Okay, so here is the simple technique.
As you increase the contrast threshold, you will be getting less false positives and you will be getting
more true positives.
And you are.
When you are getting more true positives, your precision value will continue to increase.
And when you have a low confidence score, you will be getting more false positives.
And when you are getting more false positives, then you will have low precision value.
And here we have the recall confidence curve.
Confirmed.
So in recalls we have the correct predictions, like the number of true positives divided by total ground
truth, like a ground truth annotations which include true positives plus false negatives.
Okay, the false negatives are that model is unable to predict that.
For example, if there is a cat, the and model does not predict anything.
This is a false negative because model is unable to predict that there is a cat in the image.
So in the conference code, like you can see that in recall, when we increase the confidence score,
our recall value tends to decrease.
Because if you just see the formula over here, as we increase the confidence score, our we might be
getting a true positive values.
But um, when we increase the confidence score, uh, our predictions might be missing.
Like we will be getting less predictions.
Okay, so when we increase the confidence score, our, uh predictions will decrease, like if we have
a confidence score of 0.2.
If I am getting ten predictions and if I make the confidence score 0.9, I will be definitely getting
5 or 6 predictions.
So as we increase the particular score, our true positives will decrease.
And um, we might be missing some, uh, predictions as well.
Like if there is a cat in the, uh, image and we are detecting the cat at a confidence score of 0.6,
and if I make the confidence value 0.8.
So we will not be detecting the cat in the image.
So the prediction is missing.
So we will get doing false negative.
So as we are unable to detect anything.
So when we increase the confidence stretch so our prediction will decrease and we will not be detecting
everything in the image.
And there will be false negatives like we are unable to detect a ground truth.
So our recall value will decrease.
Okay.
So as we increase the confidence threshold, the recall value will decrease.
And you can see as well at a confidence score of 0.0 we add a confidence score of 0.0.
We are getting the best recall score of 0.96.
Okay.
So now as you can see over here, if you have ten ground truth images, annotations of Cat.
And if you make five predictions, we are getting four out of five predictions.
We are getting four true positives and false negatives.
Like we are unable to detect the ground truth, so the recall value will be 40%.
Okay.
We might be getting higher precision, but the recall value will be poor okay.
So when we increase the confidence threshold, uh, many of the, uh, predictions gets missing.
And many of the when the many of the predictions get missing, then we will not be detecting all the
ground truths.
And we they will go to all negative false negatives.
And when the four of false negatives continue to decrease, increase and then the recall value will
decrease.
Okay.
So as you increase the confidence score, the recall value will continuously decrease because there
will be less predictions.
And here is the model predictions on the validation batch.
We are not using the validation data set images for the training.
So it's always better to have a look and see how our model is performing.
So we are able to detect the sitting position of the person called down as well standing as well.
So we are just getting good results over here.
So I have already trained the model and saved the model weights onto my drive.
So I'm directly downloading the model weights from drive into this Google Colab notebook.
So you can also validate your model to see how our model, your model generalizes as well.
Okay, I mistakenly run the training.
I don't need to run it, so I'll just post this.
So.
Here we are.
Okay.
So we are here.
So basically you can also, uh download the model here.
So after training I've just placed the model weights onto my drive.
So I'm directly downloading the model weights from drive into this, uh, Google Colab notebook over
here.
Okay.
So now I'm just will just perform inference on the test data set images here.
So here I'm just performing trends on the test dataset images.
And let me just plot, uh, the results we get on the test dataset images.
So this will take few seconds.
And let's.
So I'm trying to display all my results into this Google Colab notebook over here.
So here you can see that we are our results are here.
So we the person is walking, we have written and here the person is standing that works.
The person has fallen down.
The person is sitting.
So our results look quite promising to me.
Yeah.
So now I will just download a random video from my drive into this Google Colab notebook, and I will
test my model performance on this video.
So in the video we have total number of 272 frames.
And uh, we are doing uh predictions on each of the frame one by one, like the person is walking,
falling Fallen down.
Okay, so let me display the output video over here and let's see how our results look like.
So here we have the output video.
Let me just download this video and show you how it works.
So you can see the person is coming.
We are able to detect the person is walking.
And here it's fall down.
So you can see that we are able to detect that the person has fallen down and our results look quite
promising.
Okay.
So let's see again here we can see the person coming up.
The person is walking.
That's fine.
Now the person has fall down.
And our model predicted that the person has fall down.
So you can test the model on some other videos as well.
So in this tutorial we have seen that how we can fine tune the YOLO 11 pose estimation model for human
activity recognition.
So that's all from this video tutorial.
Thank you for watching.