Hi guys.
So in this lecture I will present an overview of object detection algorithms which include R-cnn, Faster,
CNN, faster R-cnn and Mask R-cnn.
We will look at the key takeaways from each of the algorithm and the primary challenges associated with
each of the algorithm.
Pursuing art objectives.
Or you can say following are the topics which we will cover in this lecture.
The first one is convolution neural network.
We will not go into much details regarding convolution neural network like what is the activation function
in the convolution neural network?
What are the kernels and what is why we use dense layer and other things?
We will be only focusing on why we use convolution neural network and why can't we use convolution neural
network for object detection?
Next, we will discuss about R-cnn.
We will see what is r-cnn and what is the primary goal of R-cnn, how we can implement R-cnn and what
are the primary challenges associated with R-cnn.
Next, we will discuss about Fast R-cnn and the primary challenges associated with Fast R-cnn.
Next, we will look at faster R-cnn hope faster R-cnn is different from R-cnn net fast r-cnn.
Why and what is region proposal network?
In faster R-cnn And last we will see what are the key takeaways from the faster r-cnn and why we need
mask CNN And in the last we will also look at mask r-cnn and its void game.
For now, we will look at the convolutional neural network.
So convolutional neural network works best if we want to do image classification.
For example, if there is a cat or a dog in the image, or if we want to do multi-class classification,
like is there a cat dog person deer in the image?
So there are four classes.
So if there even ten classes convolutional neural network works best.
Function neural network works best for image classification.
So what basically image classification is.
Let me explain over here.
For example, we have this image and let's for example, we have a cat in this image.
Okay?
So.
With image classification, we will have the output as.
Like we have that cat in the image, or if we have multiple objects in the image, we will have the
output like cat dog person.
And like this is, this is the output we get using.
Convolutional neural network for image classification.
But in case of object detection, we need to figure out where is the specific object in the image.
Like we need to find out where is the cat in the image.
So in the case of object detection, we need to draw a bounding box like this bounding box around the
cat in the image.
So why can't we use object detection?
Or why can't we do object detection using convolutional neural network?
You might be thinking this.
Okay, so if we have a single object in the image, if we have a single object in the image, then we
can use convolution neural network for object detection.
Like here we have only a single object in the image.
Here we can use convolutional neural network for object detection, but if we have multiple objects
in the image, we can't use convolutional neural network for object detection.
But if is there a way out, you might be thinking, yes, there is a way you can use convolution neural
network for if we have multiple objects in the image.
But this will not work.
This will fail.
This approach will fail if you try because we can't use convolutional neural network for object detection
if we have multiple objects in the images or video.
But if you try to do it like like I have this out listed over here, one way to do object detection
using CNN is to divide our image into smaller grids, blocks or grid, for example.
This is my image.
And here I have multiple objects like this is these are the two objects and this is the third object.
So I will divide my image into multiple grids, like let me change the color per grid.
So here I'm dividing my image into multiple grids, like this is one, two, three.
So we are dividing we have divided our image into three cross three grids.
Okay.
So one way to do object detection using a CNN is to divide our image into smaller grids.
But if we have objects of different sizes, image like we have different objects, multiple objects
like 70 objects or ten objects in the image.
And all of the objects are different of different sizes.
We can divide image into 1000, cross 2000 grid like this is we here we have divided the image into
three cross three grid.
So now we can divide the image into 1000, cross 1000 small grids, but it will definitely become computationally
expensive.
Like if you try to run this script on GPU, your system will crash, GPU will crash.
So this will not be work.
We can use CNN for object detection, but if we have a single object in the image, but if we have multiple
objects in the image, CNN fails for object detection.
CNN works best for image classification, but CNN can't be used for object detection if we have multiple
objects in the image.
Now I will discuss the region based convolutional neural network.
So let's first look at the goal of the region based convolutional neural network.
The goal of the region based convolutional neural network is to take an input image and correctly identify
where are the objects in the image via bounding box.
So let me explain you what basically is this?
For example, this is my input image.
Okay, so let me change colors.
So for example, this is my object number one.
This is my object.
Number two, this is my object.
Number three, this is my object number four.
So this is my object number five.
So I have five different objects in the image.
So the goal of region based convolution neural network is to take.
This is my input image.
Okay, so region based convolutional neural network will take this input image and in the output we
will get the objects with bounding box like through region based convolution neural network.
We can identify where are the objects in the image via bounding box.
So in the output we will get.
So output in the output.
You can have the bounding box around each object like this is the bounding box for three objects.
This is the bounding box for the four objects.
This is the bounding box for the five objects.
So one region based convolutional neural network takes an input image and in the output it correctly
and identifies where are the objects in the image via bounding boxes.
So how it works.
Input images pass while in output we have the bounding boxes plus label for each objects in the image
like this can be a dog.
This can be on the card.
So these are the labels for Cat and this can be a cow.
Or this can be a camel.
Or this can be a sheep.
Okay.
For region based convolutional neural network was proposed in 2014 by Ross Girshick and other team members.
What are CNN is called?
CNN basically work or CNN?
The secret of what our CNN does is it divides the image into a number of boxes instead of grids.
Okay.
So instead of dividing dividing the image into grids, like what is basically grid is like in we divide
the image into grids like this.
And these are the grids we are creating.
So in our CNN, we don't do this.
So this approach isn't followed in our CNN.
So our CNN basically divide the image into a number of boxes instead of grids like you can see over
here.
I just.
Just highlight it.
It is the highlighter.
So you can see over here in our and we divide the image into a number of boxes instead of grids.
So our CNN uses selective search algorithm to extract 2000 regions from the images.
So in our CNN using selective search algorithm, we extract 2000 regions from the images called region
proposals.
Okay.
In our CNN, we identify different regions in the image and then pass it to the feature extractor,
which is function neural network.
So, for example, in our CNN, we don't divide our image into grids.
So let me explain this point to you.
But look.
So, for example, here we have the image in Rcnn and this is the object in the images.
So in CNN, we will divide our image into 1000 boxes like here we can.
We have a one box here.
We have another box here.
We have another box.
Okay.
So in our CNN, we are using selective search algorithm.
We extract 2000 image regions from the images called region proposals.
And after extracting 2000 regions from the images, we pass it to the feature extractor of the convolutional
neural network.
So this is what we do.
That is why it is called region based convolutional neural network, because we extract 2000 regions
from the images which are called region proposals.
So this is how our CNN works.
What region based convolutional neural network is based on the following steps, like we generate a
set of region proposals using selective search algorithm for the bounding boxes like we.
We basically have an input image.
Then we select 2000 regions from the input image, and then we pass the 2000 regions we selected from
the input image through the CNN feature extractor to extract the image.
So basically CNN feature extractor here is a Pre-trained Alexnet.
Okay, then we have an SVM to see what object the image has in the bounding box is like now, basically.
Basically after having an in passing, after getting 2000 regions from the input image and passing the
image through the Pretrained Alexnet for feature extractor and then using SVM, we classify what are
the different objects we have inside the bounding box.
Okay, like we can have a cat dog go inside the bounding boxes and then we run the bounding boxes through
a linear regression model to output detected coordinates for the boxes once the object has been classified.
So using linear regression, we find the coordinates of the bounding boxes.
So what are the issues with our CNN like in our CNN?
For each image, we extract 2000 regions and then we pass to the CNN feature extractor.
So when we extract 2000 regions from the image and pass it to the CNN feature extractor, this makes
all the process very slow and it cannot be implemented in real time because each image takes 47 second
to test to find, to be implemented.
So if each image takes 47 seconds for the processing, so this is too much time.
So we can't use a CNN for in real time because what are the issues with our CNN?
Is that for each input image we extract 2000 regions and then we pass through the function neural network
for the feature extractor.
So what we can do is if we have an input image and we pass it to the feature extractor and then we select
2000 regions after the feature like selective algorithm or using CNN feature selector after passing
through the.
Feature selection algorithm extract 2000 regions from the images or through a selective search algorithm,
and then we can do further processing.
This can make the whole process very much better.
So here is the block diagram of the region based on neural network.
Here we have the input image.
We extract 2000 regions from the image and using CNN feature Extractor, then we extract features and
then we classify like there is an aeroplane per server monitor.
So the process can be made very efficient if we have this region proposal network after this.
CNN feature extractor like first we have the input image, then we have the input feature extractor.
Then we can have the region proposal network at the third point over here, Like we can move this over
here, okay?
And then we can classify like this is aeroplane person or TV monitor.
But let's look at the first CNN.
What basically we're doing, our CNN is that we identify 200,000 regions in the image and then we pass
it to the feature extractor, CNN based feature extractor.
So what we do in Fast CNN is that we have the input image.
Instead of identifying 2000 regions from the image, we send the whole image to the CNN feature extractor.
And then in the feature space, we have the region proposal method.
Okay.
So in 2015, the authors of our CNN came with the idea of fast CNN to address the drawbacks in the CNN
in Fast CNN.
The input image is passed to the function neural network based feature extractor to generate convolution
feature map.
And from the convolution feature map, we identify the region of proposals.
The reason why fast r-cnn is faster than our CNN is that we don't need to feed 2000 regions proposals
to the conventional neural network each time.
Instead, the convolution operation is done once per image and the feature map is generated from it.
So here we have the reasons that fast CNN is faster than R-cnn.
Is that because we don't need to identify 2000 regions from each of the image and then pass it to the
convolution neural network?
What we do is fast r-cnn is that we have the input image.
We pass it to the CNN feature map and after passing it to the CNN feature map, we get that 2000 regions
from the image.
Okay.
Block diagram of the CNN.
So this image is passed directly to the basically convolution feature map for feature Extractor.
Then we have the ROI pooling layer where we extract 2000 regions from the image, and then we have the
SVM function and then we apply linear regression like all the steps.
But instead of getting 2000 regions from the image, we pass the image directly to the condition feature
map and then we extract 2000 regions from the image.
Women are the primary challenges associated with a fast r-cnn like fast R-cnn perform better than R-cnn.
But we observed the performance of the fast R-cnn during testing time slows down while using region
proposal like what it means by region Proposal is that during testing time when we extract 2000 regions
from the image using selective search algorithm.
So the performance of the first during testing time gets down as compared to not using region proposals.
Therefore, region proposal is the major reason resulting in that degradation of the performance of
the fast r-cnn.
Okay, so here is the chart as well.
Like comparison of object detection algorithm.
You can overdo it further and see by including region proposals.
Our performance slows down while without excluding region proposals.
Our performance in testing time is better like including region proposal.
It takes 49 seconds while.
Exploding this region.
Region proposal will take 47 second for CNN.
Okay.
While with the fast r-cnn using the region proposal, it takes 2.3 seconds while without region proposal
it takes 0.32 seconds.
So using region proposals is a major reason that degrades over the performance of fast r-cnn.
Eyes are CNN and faster.
CNN basically use selective search algorithm to find the region proposals or like or you can say 2000
regions in the image, which basically slows down the processing of like we have discussed that in the
previous slide in at in testing times using region proposals using if you use region proposal, our
processing becomes very slow and it is a quite time consuming process as well.
So in faster CNN instead of using region proposal like extracting 2000 regions from the images using
selective search algorithm in faster CNN, we don't use selective search algorithm it in faster.
As CNN, we allow the network learn the region proposals.
There is a set separate layer for the region proposals in the faster rcnn so in faster r-cnn the author
made this improvement that they eliminated the use of selective search algorithm in faster R-cnn We
don't use selective search algorithm.
Why in faster r-cnn we have a separate layer for to find 2000 regions from the images.
Okay, so faster.
CNN is object detection algorithm which eliminates the use of selective search algorithm and let the
network learn the region proposal so we don't use selective search algorithm in faster r-cnn because
selective search algorithm is the major reason of slowing the processing in r-cnn and fast r-cnn similar
to fast r-cnn.
The image is provided and as an input to convolution network which provides a convolution feature map.
So in like we do in fast r-cnn is that the the workflow of fast r-cnn faster r-cnn is same like in fast
R-cnn we have an input image and then we pass through the convolution neural network based feature map.
So the same we do in faster r-cnn as well, like in faster r-cnn.
The image is provided as an input to a convolution neural network which provides a convolution feature
map instead of using selective search algorithm.
So in fast CNN we use selective search algorithm while in faster r-cnn we don't use selective search
algorithm on the feature map to identify the region proposal instead of.
In faster.
Our CNN, we have the region Proposal network.
Okay.
So in faster r-cnn we use region proposal network instead of selective search algorithm to identify
2000 regions in the image.
Okay.
Architecture of faster r-cnn As you can see here in faster R-cnn we don't use selective search algorithm
instead of we have a separate layer which is used to find 2000 regions in the image.
Okay.
So using region proposal network layer, this is the layer which point 2000 regions in the image.
So this is the difference between foster asking and the prior algorithms.
Following are the key takeaways from the faster arsenin.
Like in CNN, faster CNN, We use selective search algorithm to find the region proposals, which makes
the processing very slow, as we have seen in the previous slides.
In faster r-cnn we do not use selective search algorithm instead of faster r-cnn directly let the network
learn the region proposals, which means it directly allows the network to find 2000 regions in the
images instead of using selective search algorithm in faster r-cnn.
A separate network is used to predict the region proposals instead of selective search algorithm.
So what are the primary challenges with faster, Faster r-cnn.
Faster.
R-cnn gives the bounding boxes only, but no semantic segmentation or semantic segmentation.
We will use mask r-cnn.
Let's move towards the mask.
R-cnn.
Look at mask R-cnn.
Mask R-cnn is built using faster r-cnn.
The main idea behind mask R-cnn is to extend faster r-cnn to pixel level segmentation in addition to
bounding boxes and class label in mask.
R-cnn we have an object mask on our output.
Like for example, if we have a person A in our output which we have a bounding box drawn around the
person.
So we have a mask as well on it.
So instead, in addition to bounding boxes and class label.
Mask R-cnn outputs an object mask as well in mask R-cnn Fully Convolutional neural network is added
at the top of CNN features of CNN features of faster r-cnn which generate the mask output.
Mask R-cnn uses a trick called ROI to ROI align to locate relevant areas down to pixel level.
The backbone of mask R-cnn is resnet 101.
This is an overview of output of mask r-cnn.
Like here you can see that we have detected car.
We have a mask on the object like car as well.
Now here we have detected a person.
Now you can see that we have a mask on the person as well.
Like there is a mask.
Thus we have a label as well for the person.
Thus here we have the traffic light.
We have detected traffic light and we have a mask on it as well.
And you can see here we have also detected a traffic light and we have a mask on it as well.
And here we have the bounding box and this is the confidence code.
So this is the label, the confidence score and the mask on the object here.
Well, let's see where we can use mask R-cnn.
Mask R-cnn is used in Detectron two, so Detectron two is a framework and main model used in Detectron.
Two is mask R-cnn.
To use mask R-cnn in pytorch it requires Detectron two mask R-cnn and backbone is Resnet 101 mask R-cnn
is used for instance segmentation and U-net is a great framework and use that is used for semantic segmentation.
In the previous slide I have might have mistakenly told you that mask R-cnn is used for semantic segmentation.
Sorry for that mask.
Our CNN is used for instance segmentation and u-net is used for semantic segmentation.
For watching this video tutorial.
See you in the next video tutorial.
Till then, bye bye.