In the section on generating new samples.
What we are going to be doing is essentially taking in an input image like this one.
Adding this up with a mask.
As you could see here is specifically the boot.
So if we take out just this part and just this other part from the input image, we have this mask.
Obviously our model can generate this.
So if we take this, um, pass this into our trained um, segmentation model or former model, we can
obtain this mask containing only the boots.
And with this prompt we pass this into our inpainting model, which is available in the diffusers library
of hugging face.
We could obtain a new sample where we've modified the region where we have the, the, the, the black
boots.
So we go from this black boots to this red colored boots.
And so given that we have the input image or we could get some input image from our veil data set.
Um, we've trained our formal model and obviously from this tool we could obtain the mask.
And then we also have this prompt.
It means that it's now very easy for us to generate or edit our images.
This in this kind of automatic manner.
Now, it should be noted that the mask generated by the former model obviously will contain the other
parts of the image.
So you would expect to have pants.
So let's change the color.
So we would expect to have pants and not just only the boots.
So if we were to pick out only the pants then we would have this zone picked out, would have this zone
tool picked out.
But now that we're dealing only with the boots, that's why we've picked out only the zone.
So what we'll do now is we'll write code, um, which will take the output mask generated from the former
model and then only pick a given class, so all the other classes will be sent or will be turned to
zeros.
And then only that specific class will be, um, given a value of one.
And then we'll obviously have a prompt related to the class.
So this means that if we, um, wanted to deal only with the pants, or if we picked the pants on the
mask here, then we'll change this boots and then we'll replace it with pants.
So instead of boots we would have a red colored pants.
Obviously we could change the the color.
So we could say green color pants.
We could modify the whole prompt and obtained our desired outputs.
So this is how easy it is now to to edit our images once we have a trained model.
Now let's dive into the code and see how we could generate this mask.
When given an input image like this one, we'll start by our generate inputs method, which will take
in the input or the image path, and then a specific mask ID.
So that's it.
Now if we do label to ID or id to label let's check out all the different labels we have.
Boots.
Boots is actually this year seven.
So when running this we're going to specify seven because we want to be able to edit our input image
based on uh, the boots could check out this image 0003 see it contains boots.
Now make sure that you have, um, other specific ID matches with what you have in the image.
So that's it.
Um, there we go.
We have written this out.
So that was it.
We have our source source image which is which we can read with, with um OpenCV.
Here we have the image path and then we have the mask.
So our mask the way we get our mask is by simply taking our model which we've trained, we pass in and
then we pass in our source image.
So we pass in the source image to obtain the output.
Now um, note that we need to carry out some pre-processing.
So we need to modify this code such that we have the pre-processed source image.
Get into the model before we obtain the mask.
That said, copying and pasting out the code which we've written already.
Um, here we have our image.
Instead of sample file path, we have the image path.
Take this off and then we resize, we cast, we normalize, and then we get the transpose.
We expand um, we add the extra dimension and then we obtain the output.
So we return our source image and we also return our output.
We could go ahead and test this out.
So let's create this new cell generate generate inputs.
And then we specify that uh we specify the path.
And then the mask ID is seven as we've just seen.
So let's run this and then see what we get.
We get in this error, transpose expects a vector of size three, but inputs a vector of size four.
So let's get back to the code and see what we have there.
Well, let's first of all take this off and then run this again.
Then what we'll do now is take off this two dimensions and resize our output.
So once we get the output we are going to do the argmax to get the the the category with the highest
score.
Then once we get this argmax we resize our output and then finally we take off the batch dimension.
So now we have our resized output.
Now we could um, there we go.
We could check out this resized output.
So instead of output now we have resized output.
Let's check that out.
We now have the right shapes.
So we could go ahead now and create this mask where all the other regions are black, while only this
part where we have the boots is actually white.
So what essentially we want to do here is say all this um, region or all the other regions will take
up values of zero, while the rest, or rather while where we have the boots will take up value of 255.
So that's exactly what we want to do.
Now, that said, let's consider that we have this input as masks or as our mask, um, generated by
the model.
Now obviously here you see that all the different values take up.
Um, or we have different values except for this position here where we have the value seven.
Let's change the color.
Um, there we go.
Okay.
So all uh, all through we have different values except for this one where we have the value seven.
So it's similar to this scenario here where we could have, um, different masks or different mask values
generated by the model.
And then, um, for a given region like this region, we have the value seven.
So we just simplifying it this way.
Now what we could do to, um, take this like take this 7 to 255 one to send this to 255.
And we also want to send all the rest to zero.
So what we'll do first and foremost is add this with negative seven times um, once.
So we will have one or ones which take up the exact shape of our input.
So here if we if we do this, if we add this up like this, then what we'll be getting will be another
matrix.
Let's draw another matrix below.
So our output matrix now will be such that we will have um values or all other values different from
zero.
So here this x is different from zero.
So here we would have x.
We would have x.
We would have x x.
And then um at that position where we had the seven because we subtracted with seven then we are going
to have a zero.
So all through here we have x and x.
Let's change back the color.
Take this and then see here we would have a zero.
So all these values are both positive and negative values.
They could be positive as well as they could be negatives.
Um it doesn't matter.
Um they are all different from zero except for obviously this position where we have um, the seven
get into the code.
We're going to define a mask.
So here we'll define a mask.
And now this max will be such that we will take in our resized output, taking our resized output and
add that plus um negative the mask ID.
So it's like doing negative seven.
Negative seven does it this part.
So so this is the the the resized output here.
This is the mask ID.
And then this is some ones which take up the same shape as the resized output.
So now we will have that mask ID plus or at times um times the ones or ones like it's actually ones
like um our resized output.
There we go.
So now we take this and add with this we, we expect that to have this kind of matrix where we have
all different values except for here where we have a zero.
Now getting back to our illustration, the next step will be to give this a value of 255 and then all
the rest a value of zero.
So first things first, given that we have both negatives and positives, what we could do is we we
could multiply this by itself.
Or better still, we could.
Where this well, same square will be confusing this with matrix multiplication.
But essentially what we're saying is let's copy this.
So we just paste it out here.
Essentially what we're saying is we have the square.
Let's take this off this to off and this other two off.
There we go.
So what we're saying is we're doing an element wise multiplication.
So it's not matrix multiplication.
We take this value multiply by itself and so on and so forth.
Now obviously again all the values will still be non zeros except for here where we have zero times
zero.
But with the difference that now all these values are positive.
So x now is going to be strictly greater than zero.
So we go from x being different from zero to x strictly greater than zero.
So we would have a new a new one year space that out.
Copy this.
And there we go.
So now our x in this case our x is strictly greater than zero.
Take off this two okay.
So that's it.
Uh we now have this new matrix.
And then the next step will be to multiply this by a very large number.
So let's say we multiply this by ten to the power of ten.
So if we multiply this by ten to the power of ten.
And then we clip the output to in such a way that the max will be 255, and then the mean will be um,
zero.
Then what we'll be getting.
Now let's take this off.
What we'll be getting now that's after the clipping.
So we again we multiply this, all these values by ten to the power of ten.
So after multiplying by ten to the power of ten and then clipping such that the max is 255, and then
the mean value is zero would obtain a new matrix.
So let's copy this again.
And piece it out here.
So we would obtain this new matrix.
And this new matrix now will take up values of x would have values of x being equal to 55.
So all the values of x are now equal to 55.
And so finally what we could do is if we subtract by or each and every value here we subtract 255 from
each and every value.
Then all these values.
Now are all these values of x now become zero.
And where we had zero it becomes -255 obviously, to convert that to 255, we just simply need to multiply
by negative one.
So now we if we subtract by or we remove -255.
And then we add a negative sign here.
So now we'll have all zeros because we obviously we've subtracted um 255 from all those values.
But here we would have 255 and all the rest zeros, zeros all these zeros.
And here zero.
So that's how we can simply convert this input mask to this mask where we have only zeros.
Um, for all the, the, the known uh, required regions or for all the regions where the category we
want to work with isn't, um, represented.
So getting back to the code, we are going to implement this.
Um, obviously we've already done this first part.
So we already now have this, um, this matrix.
So let's go ahead and multiply it by itself.
So we could do our new max.
Now it's simply going to be uh, multiply, multiply by itself to make sure that we only have positive
values.
So we have mask mask.
And then after multiplying by itself we multiply by a very large number.
Let's get back to the code.
We get.
Multiply this by a very large number.
We'll have uh one e to the ten times.
Um this mask.
So now we've multiplied by that very large number.
The next thing will be to clip.
So just as we have seen here, we clip, we're going to clip and make sure that our values fall between
0 and 255.
And since obviously all the values will be greater than 255, they will come down to 255 and and only
have the zeros which will remain zero.
So getting back to the code, let's reconnect.
Getting back to the code we have NP clip.
And then we specify our mean and the mask on the max actually.
So here we have the max which is 255.
There we go.
So that's it.
So now we've clipped getting back to our explanations.
The next thing we want to do is subtract 255 and then multiply by a negative, um one.
So uh, what we could do, what we could do is simply, we could simply take, um, 255 minus this.
So let's get back and just do 255 minus that okay.
So now we have our mask.
And then after obtaining the mask let's, um, print out the mask shape instead.
Now.
So after obtaining the mask what we'll do is we'll use um OpenCV to write this.
So we could check that out.
So write our mask mask jpg.
And then we have mask okay.
So let's run that and see what we get.
Now we open up our mask and normally we should get oh we get in this instead of what we expect.
The reason we are getting this output, instead of, say, the region specific to only the boats, is
because the model doesn't perform well with, um, categorizing this region as, uh, boots.
So let's check out shoes.
Um, if we go shoes, there we go.
39.
So let's replace that ID with 39 and see what we get.
Let's say 39.
Run that again and check out mask.
Normally you should, um, get that because, uh, the model confuses the boots with a shoe.
Just close this and check that again.
There we go.
As you could see, now we have this region, um, 39, which is, um, all in white, while the rest
is in black.
So that's it for this part, we're able to generate our inputs, as we had, um, seen already from
our initial schematic.
We have this input, uh, we have the original image, and then we have this, um, region.
Now, the next step will be to carry out the inpainting.
That is to take this input image, pass it to our inpainting model, and then, um, edit our input
image such that we have this new image the exact inpainting model will be using will be the stable diffusion
tool inpainting model from stability AI and which is made available on um hugging face via diffusers
library.
So as usual, when working with hugging face, you have this um, simple examples which can help you
run the code very easily.
So here we have how it works.
You see, you can have an image, you have the mask and then you have some output.
So let's go ahead and test this out.
Um here we have some notes.
Despite not being a dependency, we highly recommend you to install Xformers for memory efficient attention.
Uh, that's better performance.
And if you have a low GPU Ram, make sure to add pip dot enable attention slicing after sending it to
Cuda for less Vram usage to the cost of speed.
Okay, so that said, let's simply copy this or we have already installed this.
So you could also copy this and install.
But we've already installed it.
So we'll simply copy this and then get back to the code.
From the code we copied.
First thing, we are going to import the stable diffusion in paint pipeline from the diffusion library.
Then we create our simple pipeline specifying the exact model ID that's from stability AI.
Then the PyTorch data type is float 16.
Um, we are going to be working with our GPU, so we have pipe to Cuda.
Then um, we specify the prompt.
But we're going to get back to this.
Now it should be noted that as we had seen, um, in this um, model card, we're told that let's scroll
down.
Uh, we're told that despite not being, uh, we recommend you installing the X format for memory efficient
attention.
So.
We are going to do this immediately after we send this to Cuda.
So let's save this.
There we go.
So that's why you see, we have this extra line right here which was not in the model card.
Then from from here we have the the image which has been generated using the prompt using the original
image.
And it turns out to be a Pil image as specified here.
Then we have the mask and we generate the images which we can now then save.
So essentially we're taking in an image.
We take in a prompt, we take it in a mask and then we generate in.
Are we editing this original image and generating a new image from our edited image.
So that said we need this image and the mask image.
So we're going to call on our generate inputs method.
Let's copy this here.
There we go.
We call on our generate inputs method.
And then here we specify 39.
There we go.
And that's it.
So now in our generate inputs method we we we output our numpy array.
So we need to convert this um into Pil images.
So that said we have image um from array from array.
That's our source image.
Let's take off the shape.
We have the image from our array.
And we also have the same image uh from array image from array.
And then we take off the shape.
There we go.
So now we have our Pil images.
Let's run our generate inputs and see what we get.
Um there we go.
Let's, let's have this image and then mask image and there we go.
So let's run this and then see what we have as output.
But before running let's update the the prompt.
So instead of face of a yellow card cat let's say we have red colored um shoe because this is a shoe
high resolution.
And let's take this off okay.
That's fine.
So we have red colored shoe high resolution.
And we're going to save this in.
We're going to save this in our let's say shoe after running.
As you could see here we have the shoe which is um, not really that red.
Maybe it looks more brownish.
Um, and then you could also compare with the original.
So you see that there's that, um, editing which we've done on the image.
Now the image size could be changed so we could resize our image so it looks like the original.
Now let's modify again.
So before we took this um the the the mask ID to be 39, let's take 13 which will present the code and
modify the code.
So let's say we have uh, a green, a green colored, um coat.
A green colored nice looking coat.
And let's run that again and see what we get.
Well, normally we should separate this so we don't have to always create all the pipeline each time
we run this.
So we'll separate the part where we create our pipeline with where we actually generate the inputs,
or rather we actually generate a new image.
So let's go ahead and paste this out here then before running here.
Now we have coat.
So we have coats.
There we go.
And then um let's run this.
So we, we already have our pipeline.
Then we, we have now this coat.
Then we're going to go ahead and resize our output.
So here we have image resize.
Um we have width init and then height init.
So instead of height width we have width height.
And that looks fine.
After resizing we could just simply display and then visualize that.
So let's run this and see what we get.
We get this output, but, um, we don't have that green color.
So we're going to modify this prompt a little here.
We say uh, photo realistic because here we were just green colored nice looking coat, high resolution
without taking into consideration the whole image.
So we have a photorealistic photo of a woman, um, wearing a green colored coat, wearing a green color,
um, let's say nice looking coat, um, high resolution and let's say all green, high resolution.
There we go.
Let's run that again and see what we get.
While that's running.
We could change this to coat.
And then we modify the code of generate inputs such that this takes in the the mask label.
And then here we'll define mask ID label to id and mask label.
There we go.
So when we pass in coat label to ID we produce the the id which in this case is 13.
So let's get back.
You see now it's much better.
We have um, um, a code now which looks much more greener as compared to what we had, uh, just in
before.
Let's run this again with a modifications, which we just had and, uh, we've already run this.
Let's run this again and see what we get.
Uh, we have our edited image, uh, only on this zone, and that's it.
So we've just seen how we could go from an input image, um, which is this input image right here to
an edited version of that image based on a specific mask, which was generated from our, um, trained
former model.