Hi and welcome back to the course in this section, we'll take a look at performing image segmentation
using some deep learning methods.
But firstly, what exactly is image segmentation?
Well, take a look at these images here.
Firstly, we've done a lot of classifiers early in the course, so classifier basically takes the entire
image like this there and classifies it into different categories of classes.
So in this case, it will say this is a dog, because most likely, that's what that image would have
been labeled, however, which we've just done in the previous lessons.
How do we figure out where the object is?
That's what localization is, and that's where object detection comes into play.
So we've done that extensively in the last sections.
So you can see this is what object detection looks like.
No, what about this?
Which pixels belong to which object?
So in this case, we want to -- everywhere.
That's a dog in a certain color or sit in class.
And we use different colours to highlight the classes in these images, as well as for a cat.
So basically, you can see image segmentation is sort of a more advanced way of doing object detection.
But instead of localizing a box, it's now doing pixel level predictions.
And that's exactly what the definition of segmentation sort of is.
It's a process of dividing an image into different regions based on characteristics of the pixels,
which means that the characteristics in this case are the classes to in order to identify objects or
boundaries, to simplify the image more efficiently, to analyze it.
So that's effectively what it is here.
So think of it as predicting the class for each pixel of an image that's effectively what it is.
So before we move into the deep learning methods, I just want to highlight that previously, before
deep learning methods became popular and in style.
There was segmentation with done segmentation was done by a few different classical methods.
And those would have been key means you can see an example of Kenyans who played a video.
You can see it's trying to find a clusters the central inside of each cluster.
That's effectively what it would do for different colors in an image.
So it tries to localize and group different colors together.
That's how Ki-Moon's would have looked, as well as is another one called means.
If looks so mean mentioned here just basically moves toward the center of mass here.
So that's how he can do.
You can do some simple segmentation with these methods right now.
However, I wouldn't recommend you use this for anything more advanced other than maybe background extraction
or foreground extraction.
So let's take a look at how Kamins works on these images here.
Suppose you want to extract a cut out of this image here and placed a cut on this background so you
can use carvings to do that.
You can see either typically how the key means us doing it here, and you can see in this one, it actually
gets where this is, where these key values this would have been too.
That gets the value it gets to cut out and then you can cut the cut and pieces of the leaf here.
This was a very old way of doing basically foreground extraction and placing it onto a different background.
But now we have a lot of a lot more advanced methods and context aware methods that we can use, which
I won't go into here because that's a different topic.
This topic is purely segmentation.
So as we can see, this works for this image, but it barely works.
It doesn't work that well.
And you can imagine for multiple classes and cluttered scenes to get something like this output where
you can actually see the cores being segmented, pedestrians being segmented, objects in or traffic
lights, trees, buildings to ruin it.
So if the sidewalk all in different colors, you might need something definitely more advanced, and
that's something is deeply methods.
So just to do a quick refresher, again, this is what image recognition tells us.
This is what object detection tells us.
And now we can see what semantic segmentation is and instant segmentation.
So take a look at these two images here.
And before I tell you what the major differences, I'm pretty sure you can figure it out.
So semantic segmentation effectively just puts all of these sheep in the same class.
Then these were sheep just making sure.
So you can see all of them are blue and they will all sort of merge into each other.
And no, the dog is in separate class as well, because the wolf or whatever he is, is a dog.
So he's a dog.
So, no, an instant segmentation.
You can see each instance of the class is in a different.
Color.
So now we can use something like an instant segmentation algorithm to actually count the number of sheep
that are being observed here, whereas in this one they would all sort of merge together here and you
don't get this separation of different classes, which is what you want and which most of these deep
learning segmentations can actually easily adapt to.
Because if you think about it, it is predicting different.
It is sheep categories.
They imagine sheep of bounding boxes for each one.
So it does have some inherent knowledge that there are three separate sheep.
However, we can actually now do the pixel wide to actually get the separating boundary, which is where
the more advanced and complicated methods come into play.
So these are more examples here.
Obviously, object detection, semantic segmentation, instant segmentation.
Likewise, you can see it in this image here below.
So now we're going to take a look at forward very popular and common deep learning segmentation models.
So those would be a signet unit, deep love, fish and tree.
There's a deep language, and one or two tree is the one that performs best, obviously, and is the
most recent.
And then there's a mask our CNN's, which are actually quite good as well.
So I'm going to go over each of these rather quickly because to get into the detail of each of these
would probably require a half full hour and you still won't be scratching the surface of it.
You just have a deeper understanding of what it is.
So I'm just going to go through the highlights of each one.
So let's take a look at segment segment was developed in 2015, and it's a semantic segmentation model.
As we know now, it does use something called an encoder decoder method.
So you can see this is how to see in architecture progressive.
As you can see, the feature maps get progressively smaller here, and then there's a constraining point
right there, and then it gets up sampled again right here and upsampling prediction.
Is this so taking this input in the training data and this is the output crunch with levels, so your
network is trying to linear mapping between this and this, that's effectively what all of these instance
or semantic segmentation models are doing.
So the network was basically based on the video G60 Network, which performs fairly well.
However, it's a very heavy parameter of a parameter heavy network, so it's not often used anymore.
So, but this is how it works.
And you can see basically, as was explained here, the rule of the decoding network is to map to low
resolution and could a feature maps to the full input resolution for Pixel?
Why is classification?
That's effectively what this is doing here.
So now we can take a look at unit and unit is actually quite similar because there's a seam up down
this basically this architecture looks at it.
Looking at here, you take the input image again, downsampling downsampled.
And there's this bottleneck constraining point and then you subsample here again soon as a contracting
path and the expensive part.
So unit was very popular and actually was very successful.
It doesn't use the video gene network.
It uses a different architecture, so that's probably why it gets better performance then segment.
But that's about it for a unit.
No.
We can move on to Deep Lab, which in tree.
So First Vision of Deep Love was released in 2014 before the others and developed which intriguing were
in 2017 and had a number of improvements.
It is a semantic segmentation model, and it definitely improves upon previous networks, as they said,
and handles the problem that deals with actually detecting objects at multiple scales, which was something
that the other networks did have some issues with sometimes.
And you can see here these are some examples of how our performance you can see it looks quite low and
it looks quite smooth in the predictions.
You can actually see it gets these horses and right correctly, and it looks quite good.
So in order to handle the problem of the segmenting, objects have multiple skills.
They had several modules that were designed to employ a truss or dilated convolution and cascade or
in parallel, sometimes to capture motor skill contexts by adopting multiple dilation rates.
So it's a bit of a mouthful, and it probably won't make much sense to you unless you read the paper
and the sentence in detail.
However, you can see again is this constricting network as we move along here and then going deeper
without the accuracy completion gives us something like this you can see to read here and how we get
the expansion going right now.
So that's basically.
Of the architecture works for this network.
Next, we'll take a look at Musk.
See our CNN's which came out in 2017, I believe and I do remember when these were released, it was
quite like groundbreaking at the time, only because it performed so well, and some of the demo videos
the researchers used were quite impressive.
I do think this was one of them with this may, maybe from detection to know what's good about the Facebooks
detection, too, is that you can actually trim and mascaras.
CNN's quite easily on your own dataset, which we'll be doing in a future lesson.
So Professor Musk are CNN's are built upon the faster our CNN's just generally deal with very, very
low.
And it wasn't hard to adapt the RCA lens to perform mask or science or segmentation effectively.
And I would highly recommend you use this one, this one and this one for your segmentation tests.
Generally, unit is quite easy to train and get up and running deep labs and treat.
Those have better performance when there's finer detail, like if you were treating a model to detect
noise and is you would probably want to use deep love, efficient tree and mascaras, Syrians generally
give very, very good performance.
I haven't had a lot of experience training them on this, on different data sets.
However, I do know it works quite well.
I was quite impressed with the training sets, the data sets that I used to treat this model.
So that's it for this lesson and what we'll do next, we'll go into CoLab and start experimenting with
some of these deep segmentation models.
And so you can actually see hands on how a unit, deep love vision tree as well as mascaras.
CNN's will be taking a look at the Facebook detection to implementation.
You can see how all of this work, so I'll see you the next lessons.
Thank you.