Now we understand in detail how the former model was built.
We are going to go ahead and create a simple formal model with hugging face.
We shall head over to the documentation where we have this, um, former um docs.
Then we click on the TensorFlow former for semantic segmentation.
Now once you click here just simply go ahead and copy out this part where we um, where we create the
model from a pre-trained one.
So let's copy this out and then um paste in here.
There we go.
Now it should be noted that this model ID um is for the former B0.
And as we have seen in the paper, the former comes in different variations.
That's from b0 right up to B5.
Um, so getting back to the code, we should let's copy this out.
Let's take that off.
Let's just call this model ID.
There we go.
So that's our model ID.
And we'll define model ID just right up here.
Model ID is equal this model ID.
And um as we've said already this is the B0.
So we could change it to B5.
And then it's trained under a D 20 K data set.
Now other parameters which we're going to pass in here will be the number of labels.
So we'll have number of labels which we're going to define num labels.
And then we'll also define id to labels.
So ID to label ID to label.
And then we'll also define label to id and then label to id.
There we go.
Finally we do ignore ignore mismatched um sizes.
Set that to true.
Now to obtain this ID to label label to id and number of labels, we'll make use of this labels dot
csv file.
So you could open that up and check it out.
Let's open this up.
You see this is our labels um CSV file.
Now let's copy the path.
We'll just copy the path and then we'll make use of pandas to read the CSV.
So we have our data frame.
Pandas read um CSV.
And then we pass in that file path.
So now we read that let's create another cell just below this.
Um there we go.
We create this other cell and then we create our ID to label dictionary.
Oops.
We have this empty dictionary.
And then we go through the data frame.
So we have for I j in iterate rows.
There we go.
Oops.
Let's get back.
We are going to say if I is equal to zero then in that case ID to label I is going to be equal Nan.
So Nan there we go.
Else id to label I is going to be equal j label list.
So we're getting this from here.
So we have this label list.
Um and this permits us now to get the exact um label.
So we're going from the IDs to the labels.
So we're matching them up.
That's it.
So now that we have this set, now that we have, uh, the cell, we could just run this and then print
out our ID to label and see what we get.
Let's run this and then say we have ID to label.
Let's see what we obtain.
See, we start from zero right up to 59.
Scroll.
Um, okay.
So, uh, we actually have 59 of them.
So it goes from 0 to 58.
So that's it.
So we have now our ID to label.
And then we have we could obtain from your label to ID.
So let's just create this um cell.
And then uh the way we obtain label to ID is simple.
So label to ID we create a dictionary.
And then we have we match the labels with the ID.
So it's kind of reverse of what we had for the ID two level.
So for id label level in ID two level items we go through each and every item, um items.
There we go.
Okay.
So that's it.
So for every item we have here we're just going to reverse.
So it becomes level.
Um, or ID takes the place or label takes the place of id and ID takes the place of label.
So let's run this again and then print out, um, label to ID.
Oops label to ID there we go.
So running that we could now see how we obtain.
Scroll down.
We see how we obtain our label to ID.
So that's it.
We've defined our label to id and ID to label.
You see we could just we could obtain this, uh, length by doing label length of label to ID and we
should obtain, uh, the length.
So here we just simply have length of label to ID or length of ID to label.
Take this off and that's fine.
So now we can run this and then um create our model.
And obviously after creating your model you could simply do a summary.
We are getting an error.
And what this error says is this model ID we specified is not a local folder.
And it's not a valid model identifier listed on hugging face.
So what we'll do is we'll get back to the documentation.
Um, there we go.
We get back here and search for former.
And you will find that that specific model isn't in that, um, exact image size.
So it's 640 by 640 instead.
So let's get back here and modify this and have 640 by 640.
There we go.
So let's run again.
And hopefully this time around everything should work.
Fine.
Let's take this off.
Run that again.
So as we're saying, we could, um, obviously do a model summary.
So we could have model summary, and we could also test this out.
So we could take in an input of shape batch by a number of channels by height by width, and then um,
see what it produces in the output.
Let's run this and see what we get right here.
As you can see we have our model summary.
It's uh 84 million parameter model.
And we also get the shape of our output.
So because we have 59 classes this makes sense.
And then also we are having 128 by 128.
So we're living from an input of 512 um 512 by 512 to an output of 128 by 128, which makes sense because
we are living from height by width to, um, height divided by four by width divided by four.
So, um, this makes sense that we have this kind of output.
And so that's it for this section.
We will now move on to the next section where we will go ahead and train um, and then evaluate our
model.
We'll start with the training and evaluation where we'll first of all create the optimizer.
So right here we pass in the initial learning rate number of training steps, which depends on the length
of our training data and the number of epochs.
Then we have the weight decay rate specified and the number of warm ups set to zero.
So let's run this and then we compile our model.
Once our model is compiled we can start with the training.
But before starting with the training we'll create a callback for our model evaluation.
So every time we offer after every epoch, we want to take in our validation data and compute the mean
IOU of the model.
To understand how the mean IOU score is computed, let's consider the following example.
Now note that mean IOU stands for the mean intersection over union.
So if you have two examples, let's let's just draw this before we get to our main example.
If we have this two boxes um then the red and then we have another one, let's change the color to blue
so you could see that clearly okay.
So if you have these two boxes then the intersection over union of them is simply going to be this area.
That's the intersection.
Let's change the color.
Um, this area right here which is the intersection of the two.
You see then divided by their union.
Their union is obviously all this part, the zone.
Um, let me change the color again so you can see that clearly.
There we go.
We have green okay.
So we take as we're saying we take all this here.
This is our union.
So all this is, um, their union.
So we take this part in white divided by all this to get the IOU.
Let's consider that here we have the target output.
So we expect, um, our model to produce exactly this output.
Here we have two different classes or two different objects.
We could say.
So let's call this object A.
And then let's call this object B this object right here.
The rest is simply the background.
So this grayish zone here is the background.
So we have object a object B.
Then we have a model uh which we'll call model one.
Then we'll call this model two.
And then we'll call this model three.
So this model one predicts this output.
And then the model two predicts this.
And model three predicts this.
Now before getting to the IOU, let's suppose that we're using a metric like the accuracy.
Now if we're using accuracy then clearly this year this model one should have the best score.
And that will simply be because we are going to go pixel by pixel and compare the actual with what the
model predicts.
It should be noted that whatever the model predicts here for this object, A is in this yellow, and
then for the object B is black.
So this is what the model predicts.
This is what the model predicts.
This is what was expected and this is what's expected.
Well this was what's expected here in the back is behind this.
So that's it.
Let's get back.
Take that off okay.
So as we're saying with the accuracy we'll go pixel by pixel and say okay for this pixel um what does
the model predict.
In this case it predicts the background and what was expected was background.
So that's fine.
Uh, we'll do that for all the remaining pixels.
And so given that the model does a great job at predicting, um, object A, although it makes some
slight error here.
So it's only this portion, this portion where we're going to have a, uh, an issue with the accuracy.
And then for this object, B the model doesn't even predict it or it doesn't notice that.
So we also have this zone.
So you find that these are the two zones where we're going to have our issues.
Or maybe this thin line here and this thin line.
Apart from this we do not have any other issues or all the rest are correctly predicted.
So our accuracy is going to be pretty high compared to these two orders where we have.
You see, for this here we have um, this zone which is wrongly predicted, this zone wrongly predicted.
Also the zone C wrongly predicted, the zone wrongly predicted.
And then here we have the zone two wrongly predicted.
So that's it.
You see that this has the smallest, um, as of all this for this tool, you see, we're going to have
this zone which is wrongly predicted because this ought to be, um, seen by the model as object A,
but it is not seen as object A.
So, um, if you use an accuracy, then you consider that all the zones are not predicted properly.
And hence this model wins.
Model one wins.
But the problem now is if you have, um, or if you suppose that this model one is the best performant
model, then you would end up with a model which doesn't know how to properly segment object B and in
the case where object um, a and B are of equal importance, then this is a poorly performing model
because it doesn't know how to to predict or how to segment.
Uh, one of the major classes.
On the other hand, the accuracy for model two will be the lowest, whereas it tries to segment a list
of both classes.
Thus, class A and class B compared to this model one which doesn't segment class B.
And so with the mean IOU, we are simply going to compute the IOU for each and every object and then
calculate the average over all the different classes.
So in this example we'll take or we'll further start by computing this area that is this intersection.
Um the intersection is this here.
So the here is our intersection.
We change this to red.
So it's clearer.
So as we're seeing here we have this intersection.
You see um there we go.
We have the intersection.
And then we have the area the, the the union here is a union.
We compute that.
See.
Compute that, uh, this this intersection divided by the the the the union.
And then we do the same for this.
Let's try to zoom in so you can see that.
So again here we go.
We take this intersection and then divide it by the union.
It's clear that in this case it does very well with the class B.
And so here we're going to have a high IOU score.
But nonetheless with this um, with this class A, we're going to have a smaller IOU score, but it's
going to be pretty reasonable.
So we could get say 0.55.
So here we get 0.55 for this with the IOU.
Um and then here we get say 0.85.
Whereas for this example 0.85.
Whereas for this example we get say 0.09.
And then here we have zero.
So if you sum this up and then you divide of let's just get for this, for this we can estimate that
we get um, an IOU because here the intersection is simply this part, this intersection.
And then the union obviously is all this.
See all of that.
So here we could get say 0.3.
And then here we could get say zero point.
Let's say 0.9 okay.
So 0.30.91.2.
We have an average of 0.6.
That's for the domain IOU.
And then here's 0.90 divided by two.
We have an average of 0.45.
So you see that the so-called best performing model uh, in terms of accuracy turns out to be the worst
performing model with domain IOU.
Then here we have 0.55 plus 0.85.
That's essentially 0.9 plus 0.5.
That's 1.4 divided by two.
We have 10.7.
So this turns out to be our best performing model.
Um, as compared to the accuracy which instead saw this to be the best performing model.
And this makes more sense because, um, all our different classes are of equal importance and so the
fact that we did well in predicting the larger object, we shouldn't rush to think that this is the
best performing model because the classes which appear smaller to matter.
That said, dive into the code.
We're going to import evaluate, and then we are going to have our mean IOU evaluate dot load the mean
IOU.
So let's run that.
See we now have our mean IOU, um, loaded very easily thanks to hugging face.
Let's define our compute metrics method.
So here we have compute metrics compute metrics.
And what this metrics takes in is simply our predictions and the target output.
So we have logits.
This is what the model produces.
And then the labels are what we expect the model to produce.
So that's it.
And then once we have this the logits will be slightly modified.
If you can see from here you'll notice that we have batch by number of classes by 128 by 128.
But what the mean IOU will need will be an output of the of type one by one, 28 by one, 28 by 59 instead.
And so that said, we are going from, uh, batch by number of classes, by height, by weight to batch,
by height, by weight, by number of classes.
So that's what we're going to permettait this.
So we'll go from zero now this two three.
So you see two three.
So we now have 223 and then one.
So that's why we take in this logits and then permutate them or transpose.
So we have logits um TensorFlow transpose.
And then we're taking the logits and then permutate and we specify it's going to be 02231231 okay.
So that's it.
So now we have this logits.
We're going to resize them because remember um the output or the target output isn't um 128 by 128.
Our target output is 512 by 512.
So because we have the stagger output and not, um, what this model produces, we will have to upsample
this.
And so that's why we're going to use a simple the resize method.
So we have our logits which is now resized.
And then we'll make use of TensorFlow's resize method.
Um resize.
There we go.
We take in the logits on on resized.
And then we specify the size the output size.
So it's simply going to be the shape of the labels because the labels are already having the shape 512
by 512.
But because we do not want to include the the batch, we're going to take one and that's it.
Okay.
So there we go.
We have we specify the method we want to have, uh, we make use of the bilinear interpolation method
for resizing.
Now talking about bilinear interpolation is simply like this.
Uh, as you can see from this paper by, um Sanjar Kashif.
Yeah.
You have an input, let's say ten, 20, 30, 40.
Then if you want to go from two by 2 to 4 by four, you see take this ten place here, 20 place here,
30 place here, 40 place here.
And then to go from 10 to 20, we simply look for numbers such that the distance or the difference between
all the numbers are the same.
So ten plus 3.3, 13.3 plus 3.3, 16.6 plus 3.3, 20.
And then we do the same in this direction, the same in this direction, the same this direction, and
so on and so forth.
So that's how we go from, um, this two by two to a 4x4 by bilinear interpolation.
So getting back to the code we're going to modify this again.
This our logits or our predictions.
Again.
Yeah we'll define prediction labels.
And then here we have argmax and text and logits resize and resize okay.
Then we'll specify the axis to be the last axis.
So negative one.
Now the reason why we're doing this is also because uh we do not want 512 by 512 by 5059.
We actually want 512 by 512.
And so we use the argmax function such that for each and every position in this, um, output, we are
going to know exactly which class um, had the highest probability.
So that's it.
We have now our predictions.
And then we could go ahead and compute the mean IOU.
So let's just say metrics uh metric compute.
And then remember we this is not actually a metric here is going to be mean I oh you mean IOU okay.
So we have our mean IOU, um compute.
And then here we specify the predictions.
There we go.
We have pred labels which we've just defined.
We have the references references.
We have labels.
Obviously from our data set we have the number of labels um number of labels.
And then we have the ignore index in the index which is set to negative one.
Okay.
So that's it.
Um, now once we are able to compute our metrics, the next thing we want to do is to obtain the per
category IOU and the per category accuracy.
So here you take this metrics.
We get the per category accuracy.
Convert to a list get that of the IOU and convert to a list.
Now let's just change this name to metric actually.
Because even though we load the mean IOU, we could also get a metric like the accuracy.
So let's change this to metric okay.
So that's fine.
Um, now that we have our per category accuracy and that of the IOU, we are going to now update our
metrics with the accuracy and IOU of each class.
Now specifying your user ID to label the specific class name.
And then we'll return the validation metrics, which is going to be printed out with the metrics method
defined.
We can now go ahead and define our metric callback.
So let's create this new cell.
And then we import Keras uh metric callback from Transformers Keras callbacks.
Let's import the Keras metric callback okay.
Then we have our metric callback which is simply going to be metric callback.
There we go.
Which is simply going to be our Keras metric callback, which we've just imported.
And then in here we're going to define the metric function we just defined.
We just created this metric function which is compute um metrics compute metrics.
And then the evaluation data set is going to be our validation data set.
We also specify the batch size which is the batch size we've already set.
So we have batch size.
Well it's actually two.
This is batch size we set right here.
So we could um instead put this just below.
So we just make use of this um batch size which has already been set.
So let's paste it out here.
There we go.
We have this paste it out and then we specify our batch size batch size which is set to two.
And then finally we have the label columns which is um simply specified as labels.
That's it.
So that's it.
We've created our metric callback.
We now define this callbacks list which is going to take in our metric callback.
So this means that you could specify other callbacks and um add them to this list with our callback
created.
We'll go ahead and recompile our model and then start with the training.
After training for several epochs.
Here are the results we obtain.
Um, as you could see at the 15th epoch, we get this loss of 0.28 validation loss of 0.36.
The validation mean I owe you is quite low.
That's 0.18.
Um, and then the validation mean accuracy is also very low 0.278.
Then the validation overall accuracy is higher compared to the mean accuracy and the mean IOU.
Now it should be noted that the reason why our mean values are very low is simply because we have so
many classes which aren't represented in our data set.
So if you look at accessories, you find that this very low value for the the validation accuracy.
Let's get to the IOUs.
Um, after the accuracies, you have the IOU for each and every class.
So we have about uh, we have more than 50 classes here.
And you see for pumps those there are no pumps.
So this makes sense that we have, um, a value of zero.
But pants which are much common have a much higher IOU score.
So the reason why we having this very low values for the mean IOU and the mean accuracy is simply because
we have this many classes which aren't represented or aren't represented enough in the data set.
That said, we go ahead and save our current weights, and then we now dive into the next part of our
work.
So we've just looked at training and evaluation.
Now we'll dive into evaluation with the 51 platform.