In some previous section, we saw how to visualize our data samples using the 51 app.
So we click on the sample.
For example you find that we have the original image and then we have its corresponding ground truth
mask.
Now what if apart from visualizing this ground truth mask with the different labels like here we have
a blazer, here we have dress and here we have a bag.
We go ahead and also visualize what the model predicts.
The way we could make this work is by modifying each and every sample we have in our data set.
So if we get our data set, head right here, we could see a couple of samples.
Then we get the image.
Using this file path we pass this image into our already trained model.
So here we have our trained model.
We obtain its predictions.
And then once we obtain this predictions we create a new key in our dictionary which we'll call pred.
So we have seen that we have the ground truth.
You see we have the ID metadata or media type file path tags, metadata, ground truth.
We'll add pred or the prediction.
And then this prediction will simply now have a value which will be this segmentation object.
That's a 51 segmentation object.
But with the difference that we are not going to create the mask path as we already have, what we're
going to create here will be a mask.
And this mask will essentially be an array which contains or which represents our prediction.
Diving into the code we are going to go for, I sample in the data set, or we want to get each and
every sample and simply modify it.
We are going to start by pre-processing the data.
So as we had seen already, or from our pre-processing method which we've seen already.
So let's take off this preprocess, take this off.
And then now we have sample file path.
See we have sample file path.
Um instead of the image path we had already.
There we go.
We now obtain the image from um our sample file path.
That's obviously after reading and decoding.
And then um we are going to also resize this image.
So here we have image resizing.
Uh we resize the image.
And uh obviously because our model is 512 by 512, we're going to have this new size or let's, let's,
let's call this um h um underscore resized and the width width.
On the score resized.
There we go.
So we have that 512 512.
That's fine.
So now we have height resized and then width.
With resized.
There we go.
So we now resize the image, and then we cast the the output image into a float 32.
Then we carry out the transpose.
Now remember our input images are.
What we obtain from our samples is height by width by three.
But what the model needs is um instead three by height by width or channel channel by height by width.
So let's we want to move from height by width by channel to channel by height by width.
And so we're going to carry out the transpose which we've already been doing so far in this course.
So we simply have img our image.
And then we transpose so that it suits what our model takes in.
And then we modify those positions.
There we go.
So once we're done with the transposition we now add an extra dimension.
We have expand dimensions.
There we go.
We have axes set to zero.
So we want to move now from channel by height by width to channel by or rather one by.
That's actually batch by batch by channel by height by width.
So that's it.
In this case it's actually one by three by 512 by 512.
Okay.
So that's it.
Uh, once we're done with that we now normalize and then that's fine.
We can now go ahead and take off this part where we had processed annotations before and then dive into
passing our inputs into the model.
So now we we we have this image.
We want to pass this into the model and obtain some output.
So here we have our output which is um gotten from passing the image into the model.
So we get logits.
There we go.
And then now because let's, let's, let's have this because when you, when you have or when you,
when you get the output, its shape is essentially the batch by the number of classes by 128 by 128.
Where where 128 by 128.
Here is simply 512 divided by four.
So because our output masks are going to be one by 128, that's essentially batch by height by width,
that's obviously the output height by the output width.
What we are going to do is we are going to select for each and every position in this, um, 2D array,
the class with the highest um probability of occurring.
So to do that it's going to be simple.
We're going to take the output which we've already had.
And then we'll pass it through the argmax or method.
So here we have output.
And then we specify the axis to be one.
The reason why we specify an axis to be one year is because it's in this axis which we are going to
be, um, selecting the classes with the highest values.
So here we have obviously zero, we have one, we have two and we have three.
So specify an axis to be one means we'll pick in this specific axis out.
So that's why we make that choice then um obviously don't forget to load the model and load it with
this um most recent checkpoints.
So now that we have this we could now go ahead and resize our image or our output.
Now remember our input images were resized into 512 by 512.
Because this is what the model takes in.
But then what we want to save is going to be a 25 by, um, 550, as we had, uh, as the image was
originally.
So we will have to resize the image and also obviously resize the masks.
Now, considering the fact that our image, which we have here is going to be of shape 825 by 550 and
also the mask is 825 by 550.
This means that we need to resize the mask, which we get after passing this image into our trained
model.
After defining the height initial and width initial, which is 825 550, we could then go ahead and
uh, make use of our image resize method.
So here we have the resized output.
Our resize output is simply going to be TensorFlow image resize which is going to take in our output
taking our output.
And then we're going to specify that the method is going to be by linear and that the anti-aliasing
is going to be set to true.
And then now what we're going to note also is the fact that this resize method takes in the the output
which.
Use one like, let's let's get back here.
We we've converted our output into one by 128 by 128.
But the resize takes in one by one, 28 by 128 by some number of channels.
So we need to modify that.
So what we do here is we're going to add an extra dimension.
So let's there we go.
Let's have expand dimensions.
Then we add that extra dimension at the end.
So we have that added.
So that's fine.
So now we we resize our image.
The next step will be to go ahead and create our prediction for or add that prediction into each and
every sample.
All we need to do is have our sample.
And then we specify the key prediction that's spread as we said already.
And now we make use of this segmentation object where we're going to specify its mask to be the prediction,
which is uh, a numpy array.
Let's not forget to specify the height and width.
So here we have height in feet and width in it.
So that's it then.
Since we've resized this into this new shape and that our output needs to be, let's say we resize this
into one by 128 or rather 825 by 550 by one, and that we need, um, our output to take this one off.
What we're going to do is we are going to squeeze this, um, output or this last axis out.
So there we go.
We have or let's, let's just do that below.
So we have resized output again resize output TensorFlow squeeze.
There we go.
Our resized output.
And we specify the axis to be equal the last one.
So that's negative one.
And then we do some casting.
So there we go.
We have cast that.
And then we specify that our data type data type should be unsigned int eight.
There we go.
So that's it.
And then once we cast this or once we squeeze and we cast our output now will be of this shape.
But again we want this shape that's we want our output to be without a batch um axis.
So we are going to take that off by simply specifying zero.
Okay.
So that's our resized output.
There we go.
We have resized output and that should be fine at this point.
Now since we're done with modifying our sample we could go ahead and save that.
So we have sample dot save and that should be fine.
Now we'll say that um, if I is greater than five we should simply break out.
So let's run that we get an error required broadcast table shapes.
Now let's get back to the code and update it so that we, um, get past this error.
Getting back to the code, we notice how we carry out normalization after the transpose.
So we should instead do this before.
Let's paste this out here and then run this again.
And with that running correctly.
Now let's go ahead and rerun our session.
After rerunning, you could see um right here that we have an additional label.
That is the prediction.
So we have the ground truth.
And now the prediction.
If we run this data set head you'll notice we have this additional key prediction.
So again segmentation object he has this ID.
And then we note how here we have this mask.
So instead of using the mask path here we use a mask.
It's the same with um what we had already with the ground truth.
So this means that if you don't want to specify the path, you could obviously obtain a mask and store
it that way in your data in your database.
So that's it.
You see the next sample we have this mask and so on and so forth.
Now we did this only for six samples.
So getting back to our visualization, you'll notice that if we take off the ground truth, you would
find that we only have this for this six samples.
Now let's click on this one here and see what we get.
We have ground truth.
You could take off the prediction or you could have the original.
So this is the original.
Let's reduce this so you could see it a bit better.
So as we're seeing here we have the original.
If you add the ground truth you can see that we have the shirt.
Let's, let's um reduce again so you could see better okay.
So as we're seeing here you see we have shared we have skin and so on and so forth.
That's the ground truth.
Then for the other prediction, you see this is what our model predicts.
So our model sees this as a shoe sees this as pants shirt skin skin.
But this portion sees us as a jacket.
Which makes sense since the shirt um, and jacket uh, look alike.
Now it has a hair.
Doesn't do so well with the hair, but, um, that's.
So we see how we could visualize our ground truth and our model's predictions side by side.
Now, if we get back to our training logs, we'll find that we had this, um, our Val or our mean IOU
scores.
We had the mean accuracy, we had overall accuracy, and then we had the per category accuracy with
the per category IOU.
So here for example, we have value for bra which was zero or belt 0.1 and so on and so forth.
So what we want to do now is visualize our data taking this output or this metrics into consideration.
So what if we want to say, um, get all the different images where our validation or rather where our
um, the belt accuracy, the validation belt accuracy is for example, greater than zero point, um,
five.
In that case, we would need to filter those out.
And and this can permit us make better decisions.
So one great thing with working with 51 is, is really easy to implement these kinds of um, strategies,
which can be very helpful when it comes to ameliorating our already existing solutions.
So just as we had seen in our section on evaluation, we have our metrics.
And then now our predictions will simply be the pred, or rather the resized output which we had from
um above.
So we change this.
We have resized output and there we go resize output, convert that to numpy, and then we put this
in a list.
That should be fine.
There we go.
We have our reference.
Uh, which is our mask.
So here is our mask.
We also put this in the list.
But then this mask, the way we can compute it is by simply, um, making use of open CV.
So open CV in read our sample we have the ground truth ground truth and then the mask path.
So we have the mask path.
There we go.
And then we obviously read this CV two in read as a gray scale.
So that's fine.
So now we obtain our mask which we pass in here.
Um the number of levels is simply 59.
So the length of level two ID will suffice.
Um ignore index zero nan to norm.
So in case we have Nan, we convert that to zero.
So Nan to num nan to num set to zero.
Um, let's get back.
Reduce levels equal false.
That's fine.
So now we've had our metrics.
We can now add a new key.
So sample mean IOU which is equal to metrics.
And then we specify the mean IOU.
There we go.
We could also do the same for the accuracy.
So yeah, we have mean accuracy.
Mean accuracy.
And then we have also overall accuracy.
Accuracy.
That's fine.
Yeah we have mean accuracy.
And then overall.
Overall accuracy.
So let's run that again and see what we get.
Diving back into our visualizations you'll notice that we now have this extra primitives.
So here we have the mean IOU.
We also have the the mean accuracy and then the overall accuracy.
So for each and every sample we have the different um values.
And so we could filter out the different samples which have a certain um overall accuracy for example.
So let's look for samples with an overall accuracy between 0.81 and 0.92.
You see that when you filter this out, when you select this, you have all these different samples.
So nine out of 77 samples, um, have an overall accuracy of uh, between 0.8 and 0.92.
Now we can go from 0.89 to 0.92.
And you see we have only a single sample.
So this turns out to be our best performing sample in terms of the overall accuracy.
So it's normal that the ground truth and then what the model predicts will will be quite similar.
You see that we have what the model predicts and then we have the ground truth.
Um prediction.
So that's it.
Now let's um, try to get that for the IOU.
Let's take this off now.
There we go.
Let's get back to the IOU and filter this off.
You see again here we have one.
We have this one.
Let's, uh.
See.
Okay.
You could notice that, um, the sample which performed best in terms of overall accuracy, is in the
exact same sample, which performs best in terms of mean IOU, but still, normally these two samples
should have their ground truths and predictions or model predictions.
Um, quite similar.
There we go.
Let's take off our prediction.
See this?
The ground truth.
And then this is what the model predicts.
That's fine.
We could check out the mean accuracy and we could also check out let's cancel this or let's reset this.
You see, you could reset and then we could also check out those with the worst overall accuracy.
So let's check out those with the worst overall accuracy.
We have five of them between 0.09 and 0.19.
Uh we can click on this one and see how poor our model performed on this one.
So here's our ground truth.
And then here's our prediction.
You see that this prediction here, it predicts all this to be pants.
Whereas the ground truth expects that this should be a romper.
Now this makes sense since, um, we shall not get the romper um, very often in our data set.
So the model hasn't learned to classify this kind of dress as a romper.
Um, for the shoes.
Here we have shoes.
Let's see what the model predicted.
Does.
It did.
Well with that.
Um, here, let's check that out.
Wasn't bad.
But what really makes this, um, or what gives this a very bad score is just the fact that all this
is a romper, whereas the model sees this as a pant and this tool.
Uh, well, this is a blouse.
So you see here we have blouse.
So all this blouse and then your pants, uh, which isn't too different from what we would expect,
but this, um, has ground.
Truth is a romper instead.
So that's it.
We could check out this other one.
Let's check out this other sample.
We take off the prediction here.
This is a blazer and the shirt.
Check out the predictions.
This is a coat and a coat sweater.
So you see um, that's why the model score is quite low.
You could check out the score here.
The mean IOU is 0.047.
The the mean accuracy is 0.27.
And the overall accuracy is 0.093.
So that's it.
Now what if we get for each and every sample the mean IOU for every um category or every class.
So this means that for a class like shoes, we could get the samples with which, um, have the worst
predictions for shoes and those which have the best predictions for shoes.
So let's go ahead and modify our code such that we don't we don't only get this mean IOU or overall
accuracy, but also, uh, a more specific IOUs for each and every category.
So now we'll go ahead and modify the code or update the code so that we will have the pair category
IOU, um, added to our visualizations.
Um, right here we'll go through the pair category IOU, which we obtained from the metrics.
So just like the mean IOU which we get, we could also get the pair category IOU.
So for let's say see in metrics we get the pair category category IOU.
There we go.
Um, if if the value of C is greater than zero point say 0.0010001, um, we are going to create a new
key because just like this, we had a key and then we had a corresponding value.
So we'll create a new key and then we'll give it a name.
What we want to have is for example shoes underscore IOU.
So that's what we want for our keys.
So here we have um ID to label.
We have k and c then enumerate enumerate.
There we go.
So that now for each and every category we have a corresponding ID.
And so now with ID to label we'll be able to get its label id to label plus underscore IOU.
So we can have for example, um, we could have shoes IOU.
And then now we have sample um shoes IOU, which is simply equal C because here we get in this pair
category IOU.
So here we'll take this off now and replace with key.
So that's fine.
Let's run that again.
And then go ahead and check out our visualization.
We now find that in addition to this mean IOU mean accuracy and overall accuracy, we have the different
IOU scores for each and every category.
So let's pick out one category let's say shoes.
Um there we go.
We click on shoes.
You see it ranges all the scores range from 0.05 to 0.8.
Now let's select between say 0.73 and zero point.
Eight.
Here we have ten out of 77 samples.
Um, get an IOU for the shoe between 0.73 and 0.0 0.8.
So let's click on this first one.
And then we could swap between the ground truth and the prediction C ground truth.
You see there we go.
Make sense.
We have here shoes.
And then when we take off ground truth.
Um although we missed out with the romper for the predictions, we have um shoes, which is um, also
which is well predicted, unlike the romper.
So that's it.
We could select another sample.
Let's pick out, say, this one.
Take out the ground truth and the prediction this original with the ground truth.
So here we have the shoes and then the predictions.
We also have the shoes well predicted.
Now what we could do also is let's take those which, um don't have shoes properly predicted.
So now it's between 0.05 and 0.16.
So there are only three samples.
So it turns out that, um, the shoes uh, in general are well predicted as compared to some other samples
which are um, less represented.
So if we, if we click on, let's say, well, just looking at this, this in general, you would notice
that most times when the shoes aren't well or correctly predicted is when the person is putting on a
clothes or a dress which covers the shoe.
So if you click on this here.
You should have this, you see.
Let's take this off.
Take off the predictions.
You see, we have just this little tip here.
So that's where we have problems with predicting the shoe.
So if you want to ameliorate, um, the shoe prediction, you would have to train on more of these kinds
of data.
Um, let's look at the prediction.
You see it picks out just this part.
Let's click on this, um zoom.
There we go.
Okay.
So you see it picks out sees this bag, sees boots, bracelets and blouse all of that.
So that's not that's not really good.
Um, if we click on the ground truth, obviously you just have their shoes.
So this tells us that with these kinds of samples, um, our model struggles to notice that this is
a shoe.
Now, if you go, move to the next one.
We also have a shoe.
Although it doesn't cover all the dress doesn't cover the shoe here.
Um, maybe because of the size of the shoe or the kind of shoe.
We don't predict all because unlike the others, where we had some closed shoes.
Um, here we have, uh, a ballerina, where you have the skin which occupies much part of the the zone
where we usually have the, the shoe.
So here we have skin according to the prediction, here we have shoes.
But he doesn't see this part.
Now, if we move to the next one because there are three samples, if we move to this next one we should
take off the prediction.
You see again here that the shoes isn't very visible.
So the ground truth here we have shoes where we take that off.
But actually normally the ground truth here should have skin because here we have some part of the skin.
So the sample wasn't properly annotated.
Nonetheless what we predict, what we predict here we take off the ground truth.
You see it predicts shorts, predict shoes, shorts.
Well, it doesn't do very well as it's supposed to do.
Um, also, as we've said already, because the trousers cover the shoe and, um, it's not, uh, close
to like, the other, um, samples we have in our data set, so.
Well, when it comes to ballerinas and sandals, um, would have more work, or we would have to gather
much more data with respect to that.
Now, let's pick out some other, let's say, hat we could pick out.
Let's reset.
We could reset here.
Could reset.
That's fine.
And then we pick out, let's say hoodie or we decide to pick out hat.
So that's it.
Hat from 0.01 to 0.38 meaning that it doesn't uh, the model doesn't perform so well when it comes to
the hat.
So let's increase this.
There we go.
We should get the number of samples.
We only have a single sample, which does a little bit better than the others.
So let's take off the ground.
Truth, take off the prediction, see the original image and then try to compare, um, the ground truth
and the prediction when it comes to just the hat.
So there we go.
We have ground truth.
See, we have hat here.
Take that off.
We have prediction and you see takes hat.
But um, see some parts of skin.
Um so that's it.
So it confused between the, the hat and the skin.
So overall we see that the, the the model struggles to predict the hats correctly though you can see
what the shoes does quite well.
Now that's it.
We have seen how to filter, um, the samples based on a specific categories IOU or based on the overall
IOU or, or rather the mean IOU or say the mean accuracy or the overall accuracy.
Now we're going to move to the next section where we shall be generating new samples.