Hi and welcome back to the course, we're about to take a look at a very cool algorithm called neural
style transfer, which allows us to copy the autistic style of one image onto another.
So let's take a look at this.
So here's an example of what I mean here we are taking an image of myself regular image, then taking
this autistic little image here and then copying that style onto the image here with me.
So you can see this image.
No copies the autistic style elements like these lines that are being drawn.
It's it's been copied all across my image.
Even my hair is kind of flake Coptic chicken in that style a bit.
So it's pretty cool.
So neural style transfer was probably the algorithm that really sparked whole II art revolution.
It was introduced by Leon Curtis in 2015 in a paper titled Neural in Neural Algorithm for Autistic Style.
Now your style transfer is basically the algorithms acronym A.T..
And it went viral after that.
I mean, there have been loads of improvements.
Loads of startups and mobile apps that aim to basically pass this ability onto the user.
So, like I said, neural cell transfer, this enables us to copy it the artistic style of one image
here onto another so that copies things like color patterns, combinations and brushstrokes from the
original source artistic image, and then applies those effects to your image.
Pretty cool, isn't it?
This one was generated by deep dream generator, and you can take a look at some of more art from deep
dream generator.
It's quite cool.
There's also a new start of wombat.
Isn't that new?
But if it's basically been pushing of this very recently, some new algorithms that are generating very,
very beautiful art, in my opinion, I'm actually get some of these and print them out and frame them.
They're they're really quite good.
So I would encourage you to check, check these sites out and have some fun messing with them.
I mean, I do a lot.
So how does this work?
How do we actually get the content?
I mean, the style onto the content image?
Well, basically, we just needed two images here.
And then what do we do?
Well, simply, we notice we take the style of one image and transmit to the other.
But how do we do it?
Well, the old style transfer uses neural networks.
What we do is we take a pre-trained network, something like a Virgin 19 or inception or arrest it.
And then we define and combine treeless functions, which we all minimize these loss functions out of
content, plus the style, loss and total variation loss.
So the first of those functions we'll talk about is the content loss and what we're doing here.
We're minimizing trying to minimize the distance in content between the generated image and content
image.
So what are we doing exactly?
Well, the content loss function measures how similar the generated image is to the content image.
It uses Euclidean distance that's the L2 norm to measure the difference between features of the content
image and the generated image.
So we're looking at features here and comparing it via Euclidean distance.
Then we just build this last function by using the pre-trained network like video g 16 or 19.
And then we select a higher level layer that serves as our content flawlessly.
And the reason we use a higher level layer is because the higher level layers have more structure and
you're able to capture content better than the lower level is.
We then compute the activation of this layer so that activations here mean that we're looking at the
filters and those layers.
And if something activates both in content and style image, that tends to mean they're similar.
So we just compute the activations for boot and we compare them to us using this loss function and we
try to minimize the distance between them.
So generally, what is this seeing is that our content image or generated image has to look similar
to our content image.
That's what this loss function is doing.
So as you can see above, we just take the L2 norm between these activations there.
All right.
So now let's talk about the style loss.
So the star loss is quite important, and what it's doing is that it's measuring how different the generated
image is and style features, by the way, from our style image.
So how are we doing that?
So this is a bit more complex and this need something called the ground matrix.
So we'll talk about that shortly.
So stylus uses multiple layers.
And the reason we use multiple layers is because we need to preserve some multi-skilled representation.
And this allows it to capture low level features.
Things like edges in the style mid-level features such as blobs and queues, and then more high level
complex patterns and stellar representation is given by something called the ground matrix.
So we'll take a look at that shortly.
However, just imagine that these tiles.
Imagine what style is.
Tell things like brushstrokes of different color patterns.
All of these things are captured in the features here.
But we now need a way to measure the correlations between those.
So let's take a look at what the Crown Matrix gives us.
So the Crown matrix for our stylus, unlike the content loss, we just don't need to find the difference
between the activations.
We need to do something more and that something more is finding the correlation between these activations
across different channels of the same layer.
So what does that mean?
Well, imagine this is the width of the network here, and these are the filters.
Which of a filter and what the gram matrix is doing is that it's measuring the correlation between filters.
So you can see looking at the screen filter here, we can see in the graph below the graph, but on
a matrix form.
And what after we complete the correlations, we can visualize it what it is between each of the filters
here.
So why is this important?
Well, remember, remember, each filter activates upon seeing an example of a feature such as like
a cat's nose.
Now, if another filter in that layer activates upon detecting a cut, that would mean those two filters
correlate.
So that means very able would able to detect a cut.
This is important in building now style knowledge.
OK, so let's talk a bit more with the Graham Matrix.
So in order to capture the style, but not the global arrangement of the content to remember, the global
arrangement is more the layout of the image and what that's what's represented in a content image.
That's what we want.
That's the actual picture.
We simply just want to get to copy style like brushes, paint strokes, artistic elements.
So to do that, we must rely on the correlations between our filters.
And to do that, we take the L2 norm between the ground matrix between layer activations.
We minimize loss between the style of the output image with our original style image and thereby forcing
the style image of the output image to correlate with the style of the style image.
Now that's a bit of a mouthful.
I read it out and I hope it made sense to you.
Take your time and read this paragraph over.
If it doesn't, just pause a video, I take a look, but I'll try to explain it to you one more time.
So what we're trying to do is minimize the loss between the style of the output image and the style
of an original style image that Lewis is represented in artistic style, which is what we talked about
in Star Wars as well as content.
So we're forcing the output image to correlate with the style of the style image and also correlate
with the content loss with the content image as well.
So that brings us to something called total variation loss now that a variation loss was included as
it reduces noisy and pixelated outputs.
It allows us to maintain some smoothness and spatial continuity, so we get nicer looking images.
And the total variation loss considers both the content loss and style loss as well.
So it operates on the output image only, with the only goal being to enhance that image itself.
So this is what the equation looks like, and we can assign a widths to the content and style.
So if you want more style, less content, you can adjust as we take respectively, or vice versa if
you want to increase the content and reduce the style just to make this bigger than this.
So thereby, we have all these these tree loss functions, so we now need to combine them.
So the final loss function is basically the sum of all previous lowest functions with each component
weighted.
This weighting allows us to tweak the results image.
So if you wanted more content and lost style, they can adjust ensuites appropriately and to implement,
we start iterating for a preset number.
So we're setting epochs beforehand and then using gradient descent, sometimes using the L Dash VFX
algorithm.
We minimize our lost functions.
So that's it.
That's basically neural cell transfer, in a nutshell.
It's this is a deep topic in the research paper is actually very tough to read.
It's it's quite complex, and implementing in the code is also quite complex.
However, will now be doing that in both Keras and PyTorch, we'll run through the court and you'll
be able to train it.
Sorry, not Trina, but you'll be able to execute it and implement it on your own test images.
So it's pretty fun.
It's basically like deep stream generator, but with your own cord now.
OK, so I'll see you in those lessons.
Thank you.