Hi and welcome back.
So what we're going to do.
We're going to implement neural cell transfer using cameras as well as using the models.
Hi and welcome back.
What we're going to do in this lesson is we're going to use cameras to implement neural cell transfer
as well as use the TensorFlow hub models to implement neural cell transfer, which is quick and easy.
So let's take a look.
So open this notebook, which I've already opened here.
And what we're going to do is we're going to set up and little modules for this and create some hope
of functions.
Then we're going to implement fast style transfer using the TensorFlow hub.
And then we're going to implement, implement our model from scratch, build a model, extract style
and content, then run gradient descent on it, and then also implement one with total variation loss
and then rerun the optimization to generate that output.
So let's take a look at this guide.
So firstly, we just import and set some environment variables here.
So this just tells us we wanted the compressed format of TensorFlow hub models that we're loading.
So TensorFlow hub, by the way, is a repository of models that can be created, TensorFlow models that
can be loaded and reused.
Anyone could for experimentation.
So it's quite it's quite myself.
Next, we're going to just import some libraries, as well as some of the plot lab plotting parameters.
Let's do that.
Also, we're going to use this AI Python display.
This allows us to display images in these notebooks.
However, if you want to display a new image, it overrides the whole image.
So it's quite convenient when you want to update images in real time.
So it's it's nice.
So now we create a helper function here that is converted tensor to image and returns the image and
still image from a format.
So let's take a look and know.
Download some of our test images.
So one of these images to Labrador, which is my Labrador, is his name is Samuel.
With the Chocolate Lab in it, we're going to use him as a content image and then we're going to use
this art painting called The Wave to input to convert style.
And then also, you have few other options here.
You can have music and is available as well.
So let's ludus images create a function to lower the image and then basically convert it into the format
that we want to use within our own networks.
Next, we created a image.
This function is to show the image as well.
And we just load the image from using this little image function that we downloaded, and then we just
display it as well with the image.
So let's take a look at those test images.
You can see my Labrador Samuel with his little reindeer antlers.
This was taken last Christmas 2020, and it was still image since it's called the wave, as you can
see why it's a pretty nice painting.
So we're going to put this style onto this picture might, might be kind of interesting.
So we're going to look at a model first from TensorFlow Hub, and you can check it out here.
TensorFlow Hub, if you want to take a look, haven't been on the site in a while.
This is the model for image stabilization.
So you can go to image style transfer and take a look at many other models they have available.
They've got six right now, including cartoon Gan, which is pretty cool.
So let's go back to our code and let's see how we how we use those models.
So to get your model, you just send us your URL here.
Now, where do you find that URL?
Let's take a look.
So.
So go back to TensorFlow hub and look for the model that you want to use.
That's we're using image stabilization vision one Dash 256.
And what you do, there's some example code and how to use those models.
This is a URL to download a model.
So when you want to bring it in to your collab or your local machine, you can just download it using
hub that it right from the URL here.
So let's head back to our notebook, and all we do is we lower the Model S and model.
There's no hub model.
And then we lowered the content image as a as a TensorFlow constant.
Similar for the style image.
And then we just extract the first element and read this returns is a stylized image.
And then we just use Tensor to image here because this returns a tensor and we have to convert it back
into an image.
So let's run this and it will give us this nice output that will be generated very quickly.
You can see this is quite fast for seconds, and we got a very good implementation of this, to be honest.
This is pretty cool.
It's much better than the all the earlier versions of neural cell transfer.
So it's a good model, and I would encourage you to maybe mess and try the other models on TensorFlow
hub.
There's six of them available here.
So let's take a look back at now implementing this from scratch, know this is going to be a lot more
difficult and the court is a bit confusing, I must say.
However, I'll try to step through it at a high level so you understand what's going on.
So first of all, let's load of a pre-trained network.
We're going to load in 19 and take a look at the layers and the network so we can print them.
And then what we have to do, we have to define.
After that, we have to define content and style representation.
So we'll take a look at how we do that.
So let's do this, which I've already done and print the mean layers here.
These are the 19 layers.
Well, actually, it will be 16 layers in this.
I believe I have the confidence because it doesn't include the top and the top of the Regas Tree fully
connected layers.
So this is where we define the layers we want to use for the content.
So remember, for the content, we need to use layers that are lower down into the network.
So we're using this one to kind of it.
We can use country or conforming Block five, but we choose to use one of two because it should keep
most of the style in this layer here.
And then we have this tile is actually when is it still in and the global localization, not the actual
stuff.
Sorry about that.
So the style is all from different conflicts.
Here you can see we took the first layer from Block one block to block three and form Block five.
So then we just get the lengths of the number of layers here that we will use in our functions.
So now, once we chose those intermediate layers, my style and content representation, we can now
build the model.
So I'm not going to read all of these notes, these notes and more for you guys to run this notebook
and do it on your own.
It's a bit exhausted for me to keep reading off this one.
This is a practical lesson, so it's a little more fun just to go through the code and then you can.
If you want to get a deeper understanding of what's happening and my explanations, my high level explicit
explanations, don't cut it.
Take a look at the notes and comments in the code.
So now we use of do we build a model here?
So this is a simple function here that just takes those layers.
So we know the network here set it to trainable to fall, so it frees up the network weights.
Then we just get those outputs here and we build back the model right here.
So this just returns the Vigilia as a model.
Before that, we were just loading it and experimenting.
It's here to just to visualize it for tutorial purposes.
So now this is the actual function that does it for us.
Next, we have to basically get style layers extractor and then the style outputs from this.
So let's do that now, and let's take a look at the output so you can see.
Let's take a look at this style is and still.
So what we're doing, the name of two layers here block me and the output, which is a outputs here.
This is the output size of that layer.
So we need to capture the output style output size of each of those interim style is just because we
need it for the network.
So let's take a look and see how we calculate style because those things are needed for these these
lost calculations.
So now this brings us to the stylus.
So remember, the stylus basically computes the correlations between filters of the pre-trained CNN
s or G 19 network.
So to do that, remember we talked about the Gram matrix to Gram Matrix is how it how we calculate those
correlations.
This is a function that does it here.
We use this TensorFlow function here called linear alg insome.
And this basically allows us to calculate that.
So let's run this function.
Remember to run each block of code because I've run this entire notebook before to just double check
that everything was working.
So you may need to, I mean, sometimes skip running it accidentally because I know Auth it has been
run still.
But just remember when you're doing it to run all code blocks.
So, no, we extract the style and content from the networks that we loaded.
So those above this here, we would just sort of experimenting to see how we can extract the different
layers and get the size and the mean and max of the pre-trained feature weights as well.
Now we just have a class that basically does all that for us.
So we have a stellar puts the content like the style dictionary and it returns everything here, the
content dictionary, Estelle Dictionary that we can see.
This is what we created up here, basically.
So that's an example of it.
And we just inherit from this class here in this function.
So my style content model and we just created like a DVD model as well, said the way it's defaults.
So this does a lot of the heavy lifting for us here and organizes the code quite neatly.
So run that block of code and we just use that.
Content.
That's our model.
Sorry.
Plus, to get the extractor and then we can run that on the content image itself and cause of the results.
So we have that now, so we have all of those calculated for the input image.
Next, we can run gradient descent here, so we have our stealth targets in our content targets that
we extracted.
Next, we just love the image here and to this from our content image and converted to a TensorFlow
variable.
Then we just clip the outputs from the outputs between zero and one pixel values.
And then we set a optimizer to be Adam with a fairly large loading rate of point zero two.
But you can experiment with these values.
And now we have a content and style weighting loss.
Note this is not the way it's the parameters the alpha and beta parameters that go into the total variation
loss.
This implementation, this basic implementation we're doing here, does not include the total variation
loss loss.
It's just loss and content loss, as you can see here, for a total loss.
So let's run that as well.
And now we can do a function to do a training step.
So we're using the t.f gradient tip to just do two steps similar to how PyTorch does it.
And now we can generate our final output and you can take a look and it looks pretty good.
This is after one epoch of running, so now we can do this, we can see the steps we took to 100 and
trained for 20 bucks of notice.
It doesn't get much better at what it would have, but you can see after 100 steps, it produces this
image.
That's the image we produce above.
But then it's it's not really improving the image, unfortunately.
So on and doing something wrong, this is the best we can get.
But anyhow.
Now let's introduce total variation loss.
So to do that, we have to set a high pass for our.
For artifacts.
So that's what this function does.
Then you can actually just visualize those Haifa high frequency features so you can see what they look
like on the image.
So these are the horizontal filters, the vertical deltas, vertical deltas with style image did horizontal
surface of the style as well.
So you can take a look.
These are basically the artifacts that we're going to try to smooth and reduce.
So now this is using the Sobol filter to basically get the Sorbo edges.
That's an algorithm that looks at the edges here.
So we used it a bit to visualize that the high frequency mapping that you see above is basically an
edge of the detector in case you didn't notice.
That's what's going on here.
So just this observation was noted here.
So now we just create a function here that allows us to get the total variation lost.
That's the high pass X and Y deltas.
And then we use this TensorFlow function.
We use some to get two absolute values of those of those outputs from that pass filter.
And basically, if you run that function on the image, this is what it looks like.
I don't know.
I have this line twice and just delete it.
And it's the same, not death anyhow.
Now we can rerun the optimization, so let's choose a weight for total variation loss.
And we used to go as the default width and we create a new trailing step function.
Because this step brings up function now has a total variation loss in it.
So we can create this here and an image as a TensorFlow variable.
And now we can rerun the optimization.
So let's take a look at it.
You can see after 100 bucks, it looks quite good.
And you can see after each e-book, it's getting better, actually, it's getting more.
It's getting to be you can see it's blending between the images now, so it's smoothing out those artifacts
that we want the total variation loss to effect.
So it's definitely doing that and it's producing a more static image as we run our e-books.
So maybe it'll finish shortly, but you can experiment to the edit finish after 23 seconds, but you
can experiment with this and try different combinations of this.
Wait, here was increased epochs if you want to actually consider wait right here, the variable again
over here.
So if you wanted to save that image, you can just run it, run this block of code and it automatically
downloads that file for you.
I can see I downloaded it before it blocked my download.
When Q&A happens, sometimes they just allow it and you'll be able to download your stylized image output
here.
OK, so that's it for the TensorFlow and Keras implementation of neural cell transfer.
Now we'll take a look at the PyTorch implementation, so I'll see you in the next lesson.
Thank you.