The whole world is a zero shot object detection model, which enables us to do object detection without
requiring the need to train the object detection model.
In YOLO world, we only need to define a prompt as a list of class names that we want to detect in an
image or a video frame, or on a in a complete video.
So one of the limitation of existing zero shot object detection models, which include grounding Dino
is the speed.
Or you can say latency.
Latency is basically the time taken by the object detection model to do object detection on an image.
So, uh, the existing zero shot object detection model takes quite some time to do object detection
on an image or on a frame of a video.
So in YOLO world, this limitation has been addressed.
So YOLO world was designed to solve a limitation of existing zero shot object detection model, which
is speed or latency.
The existing state of the art object, uh, state of the art zero shot object detection models such
as ground in Dino use transformers in their architecture, which is a powerful architecture but a slower
architecture.
YOLO world in YOLO world architecture.
Watch this.
CNN based YOLO architecture is being used according to the paper or according to the YOLO world paper
that the team has published.
YOLO world achieves a mean average precision of 35.4, with 52.0 for the large version and 26.2 average
precision, with 74.1 for the small version.
While on the 300 GPU and Nvidia 300 GPU, which is a powerful GPU.
Um, achieving such high FPS, which is around 52.0 fps with the mean average precision of 35.4%, is
very impressive.
So as I told you in the hello world, you only need to pass the class names as a list into the input
prompt.
So now you can say that, uh, I just if you want to, if I want to detect the, uh, person in red,
I just need to write in the prompt the person in the red, and it will offer you the world model will
reject this person, which is in red jacket.
And if I want to detect, uh, uh, the brown animals in an image which has many different animals.
So I just need to pass the in the input prompt, the brown animal, and it will reject the brown animal
in the image.
And if you want to detect the tallest person in the image, you just need to pass the input from the
tallest person and it will detect the tallest person in the image.
Similarly, if you want to drink the person with a white shirt, you just need to pass the prompt as
a person with a white shirt and it will detect a person with a white shirt.
But before we run the script, uh, let me, uh, let's make sure that we have selected the runtime
as GPU.
So we have selected the runtime as T4 GPU.
So if you just go over here, you can see that we have selected t4 GPU.
So in this tutorial we will see how we can do object detection or image or on a video using YOLO world
model.
So let's run this cell.
So this will just make us sure that we have selected the runtime as GPU like GPU memory usage.
And that's fine.
So now, uh, to, uh, manage dataset images and models, let us set the our own uh, directory as
home.
So now we will install the required packages.
So we will be using two packages with the two main packages which we will be using.
There are other packages or libraries as well but that there are two main packages are Python packages
that we will be using this tutorial.
One is inference and other is uh supervision.
So these are uh both are available under Roboflow.
So inference for zero shot object detection uh using YOLO world and supervision for post processing
after doing zero shot object detection on image or on video.
So as I told you, both of these packages are available under Roboflow.
So Roboflow inference is an open source platform designed to simplify the deployment of computer vision,
uh, models.
So it enables developers to perform so using inference we can uh perform object detection classification,
instance segmentation.
So using uh inference you can use YOLO v9 or Yolo V8 model YOLO world model.
So these are different models that you can use in in a single package.
So YOLO world or inference package enables developers to perform object detection, classification and
instance segmentation.
Uh, like you can use different segmentation models which include segment and anything model as well.
Plus our YOLO world is also available under this um Python package which is inference.
Okay.
So uh, to do object detection image or video, we will be using um inference package so that we can
do object detection on image or video using your world and to do post-processing on the results.
Then we will be using the supervision package okay.
So let's install this package.
So the package installation will take quite some time.
So let's wait for this package to get installed and then I will go back.
But the inference package is being installed.
So next we will install the supervision package.
So this will take a few more seconds before this package will get installed.
Okay.
So okay so this package is being installed.
So we require supervision package for post processing after we have done detection using uh inference
package uh using that YOLO world model.
So now I'm also ensuring that you don't package because I will be downloading as sample image and a
video from my Google drive.
So to download a sample image from or an video from Google Drive directly in the Google Colab notebook,
we required that ground package.
So next we will be importing the required libraries.
We require the OpenCV Python library so that we can draw bounding box bounding boxes.
We can add text or all this stuff.
Then we require supervision package for post-processing.
Then we require math uh package.
So our libraries so that we can calculate the confidence score.
Then we also require Tqdm library.
And then we require the inference library so that our inference package so that we can do object detection
on image or video using your world.
Then to display an image in the Google Colab notebook.
Uh, we require the IP from IPython dot display.
We require the code image package.
So now I'm downloading a sample image.
And our two demo videos from the My Google Drive directly into this Google Colab notebook.
So I've added the image and this window videos into Google Drive, and I'm just adding the link over
here.
So I will be able to download this image and these videos from my Google Drive directly into this Google
Colab notebook.
So let's run this two cells over here.
So now you can see that we have, uh, downloaded a sample image.
And now I'm also download trying to download two demo videos so that I will be doing object detection
on this image and video using your world.
So now I have just set the source image path as this image which I have downloaded.
And you can have just set the source video path as one of the video which I have downloaded over here,
which is test or.
So the underworld comes with three different models the underworld Small Model, Underworld Medium Model
and Underworld Large model.
In this tutorial I will be using the Underworld large model, but you can try out with your old medium
and small model as well.
Okay, so as we are using inference package which are which is available under Roboflow.
So to use the underworld model I'm using inference package.
Uh so which is available under Roboflow.
But we don't require the Roboflow API key.
So that to use the uh, your world model.
Okay.
So now you can see I have just initialized the, uh, or I just loaded the YOLO model world model over
here so you can see that we have, uh, gradient instance of the YOLO world class over here.
And, um, the, the model ID, I have just initialized the YOLO world large model, which I will be
using in this tutorial.
Okay.
Uh, so here you can see that we have loaded the model with the world model by simply creating an instance
of your world class.
So this pseudo world class has two basic core functions.
One is the set classes and one is the inference.
Okay.
So in the previous zero shot object detection models which include uh grounding dino as well.
So in that in this zero shot object detection model, real time text encoding is being done.
But in case of the world, we do not do real time text boarding because the world utilizes the prompt
that then detect paradigm.
In the hello world, we first, uh, pass the prompt and then we do the detection.
So we do not do real time text encoding okay.
So you can see over here in that classes, I've just created a list of different objects that I want
to detect in my input image okay.
So.
So you can see over here this is my problem.
So this is you can say that this is my input prompt which I'm passing over here okay.
So when I just call the set classes method in this hello world class we have the set classes method
as well.
So uh, when we I call the set uh classes method over here.
So basically this input prompt which is the classes over here are being encoded into offline vocabulary.
So when I cross call the set classes method.
So this uh these classes will be converted into offline vocabulary okay.
So we don't need to perform real time text encoding.
And this uh basically speed up the process as well.
Okay.
So now when I run this cell, you can see that or this something is being downloaded.
So basically, uh, the clip model is being downloaded in the background, which convert this to the
list of classes into a embeddings.
So we are just creating embeddings in the background okay.
So now we need to perform object detection on the sample image using this YOLO world large model, which
we have initialized over here.
Okay, so as I told you in the start that this hello world class has two core functions.
One is a set classes.
So which we have used over here.
The set classes basically convert the uh our input prompt, which is the class names over which we have
defined over here into embeddings.
Okay.
Or you can say we have created an offline vocabulary.
Then we have a second method or second function which is infer okay.
So this is not inference.
So this is inferred.
So.
So now we just pass the loaded image to our inferred method.
The second method which is inferred.
And using this we will be able to do object detection on that input image okay.
So now you can say that I'm just passing the loaded, loaded image to the second method, which is inference.
And that's, uh, draw the bounding boxes around each of the detected object using CV two dot rectangle.
And then I will be adding the confidence score and label name as well.
Label.
In the label we have the confidence score and the class name.
So let's do all this.
And to save the output image I'm using cv2 dot imwrite method okay.
So okay so my output image is being saved by the name output dot jpg.
So you can see we have the output dot jpg.
So let me display this output image over here.
So now you can see over here.
In the list of classes which I want to direct.
I posses the handbag, shoes, bicycle, purse, traffic light, backpack and purse on.
Okay.
So it has directed a handbag over here as well.
Okay, well it has shoes as well.
Bicycle as well.
But you can see over here there are two persons.
This person and not detected.
This traffic light is not detected over here as well.
So, uh, lets us, uh, by default in the other world model, the confidence score is being set to
0.5.
So we will now, uh, like, you can see that now that many classes from our programs were not detected,
like you can see over here, these shows were not detected over here as well.
And, uh, these persons were not detected.
So, uh, there are many different classes that were not detected.
Okay.
Like this traffic light is not being detected.
The background traffic light.
So we will just, uh, decrease the confidence score.
So the by default, the inference confidence threshold is being set to 0.5.
So we will decrease the confidence score to 0.10.
And let's see if all the objects are detected okay.
So and let me just show you the output image now.
So now you can see over here we have predicted this shows as well.
We have detected this traffic as well.
But uh, we have detected these persons as well, these persons as well.
So the results are pretty impressive.
But we are just facing one other issue, which is duplicate detection or double detections.
Like, you can see that there is one bicycle, but the bicycle is directed two times like you can see
here.
Bicycle with confidence score 0.15.
Bicycle with confidence score 0.43.
Okay, so, uh, they are double reduction, like there is one bicycle, but the bicycle is being detected
two times.
To solve this issue we will be using non expression okay.
Or now you can see over here we have the issue of double detections.
Like for a single object we have two bounding boxes one with confidence score 0.43 and other with confidence
score 0.515.
So now you can see that to uh remove uh, double detections or to eliminate double detections, we will
be using a technique called norm expression.
So norm expression is a technique which is used to remove redundant or overlapping bounding boxes.
So now you can see that we have two overlapping bounding boxes for this uh with one with bicycle with
confidence for 0.43 and other with confidence for 0.15.
Okay.
So non-maximum suppression is a technique which is used to remove overlap bounding boxes over here.
So over double uh bounding boxes like you can say for a single object okay.
So the norm expression we have uh threshold uh which values varies from 0 to 1.
Okay.
So if uh, we set the threshold value high, this will result in fewer bounding boxes being suppressed,
uh, which may lead to more overlap, uh, bounding boxes.
So.
If we set the threshold value high, like the threshold value that in norm expression we have a threshold
value.
So the threshold value ranges from 0 to 1.
And if I set the threshold value for example 0.8.
So this will result in more overlap in bounding boxes.
But if we reduce the threshold value like I set the threshold value 0.1 over here.
So you can see over here I have set the threshold value 0.1.
So the lowest threshold value will result in more aggressive detections okay.
Wouldn't be a treasured value will result in a fewer but more accurate detection.
So we have just set the low threshold value, which will result in, uh, low, uh, fewer but more
accurate detections.
And we have set the confidence value 0.10.
Okay.
So let's run this up now.
And let's display the output image over here.
But now you can see that this issue is being solved.
Like, you can see that, uh, here we have been facing the issue of double detections.
So, uh, now this issue is being solved over here and we are not facing any such issue of double detections.
Okay.
So using non-max suppression, uh, by setting the, uh, higher values, uh, value of, uh, non-max
version, I just set the threshold value of, uh, fewer and lower threshold value, which results in
more accurate detection.
And I was able to eliminate the double detection.
Okay.
Or I was able to remove the overlap bounding boxes by setting the threshold value to 0.1 or 2 three
lower threshold value.
So now.
And I will be doing the processing on the video.
And the class which I want to detect in the video is backpack.
So I want only want to detect the backpack.
So the persons will be wearing backpack.
And I only want to detect the backpack.
Okay.
So let's run this cell.
So now we are just converting our this input prompt, which in which I have defined the class name into
encodings over here.
And now I'm just doing the same process over here, which I am following above.
Uh, but just I'm just waiting a while loop and we are doing detection on each of the frame one by one.
So now you can see the frame count over here as well.
And uh, you can see over here the different uh, dates, which is we are only detecting backpacks.
So now you can see that in each frame around 3 to 5 backpacks are being detected from the results,
which I can see over here.
So I think this video has total around three.
This video is being split into around 341 frames, and we are doing detection on each of the frame one
by one.
So okay, it's done.
Now let me display the output video over here.
So now you can see that our output video is being saved by the name output dot API.
Because I have written the name that output video name will be output dot Ava.
And we are just uh, like 30 frames over here.
So, um, very soon we will be able to display the output video over here into our Google Colab notebook.
Well there is our output video.
So let me download this video first and let's see.
Uh, let me show you the results we are getting.
So here is our output video.
So now you can see that we are able to detect the backpacks okay so the results look quite good.
Like we are able to detect the backpacks over here.
So in the same way if you want to detect like the shoes or the persons uh, you can also do this by,
you can simply pass the names of all those objects that you want to detect over in this list, and you
will be able to detect the persons shoes or, and all these stuffs.
So that's all from this tutorial.
Thank you for watching.
Bye bye.