Hi there and welcome to this new session in which we are going to delve deep into generative adversarial
neural networks.
In the previous session, we have seen how we made use of the Variational auto encoders to generate
these kinds of outputs, which are in fact mixed digits from just an input noise.
So what we had was this kind of encoder decoder structure where we had some input images here, like
this one, for example, some inputs, and then we train the model such that the outputs look like this
inputs.
And then once training is done, we could take off this encoder and then just pass in a noise signal
in here and then generate new.
Outputs like what we have here, which look like those from the original data set.
And in this section, we'll be seeing how to build a new set of or a new category of generative models
known as the Gans.
And we'll use them to generate images like this one, which we have here.
You should know that all those images are images of people who do not actually exist.
But before we dive into practice and see how to build models which can build this kind of realistic
looking images, we shall start by understanding how this generative adversarial neural networks that's
Gans work.
Now this Gans, we're first introduced in this paper by Goodfellow et al, where the GAN architecture
was first proposed.
And to understand how the Gans work, let's make use of this figure from Luis Bouchard's post.
Now let's suppose we have the bank which produces real money.
You see here we have this real $100 bill.
And then on this other end, we have the thief who produces fake money.
You see here, this hundred dollar bill has this man with a moustache, which is in the case of the
real dollar bill.
And because differences like this and say, for example, this compared to this are very clear or can
be easily seen, this police officer is able to detect that this money is fake.
But when this police officer detects that the the bank note is fake, he or she says that it's fake
because we have a mustache, for example, because we have this word fake written on it, because this
hundred has a fake around it and stuff like that.
So it tells the forger or the thief what needs to be ameliorated in order to make sure that the next
time this thief presents this fake money to the police officer, the officer thinks it is maybe this
real money.
And so if we suppose that real money takes a value of one and fake money takes a value of zero.
Let's change the color.
Fake money takes a value of zero.
Then let's say in the first year of production of this fake dollar bills, the police officer correctly
says, okay, this is a zero and this is one.
That's all it's very evident at the beginning.
But then with time, this thief gains experience and now produces fake money, which looks just like
the real money.
And this now pushes the police officer not to be able to distinguish between the real and the fake any
longer.
Now, although we do not advocate for these kinds of malpractices, it turns out that this is the way
the gangs actually work.
And in our case, we'll replace this generator or we replace this thief by a generator model, which
is a neural network.
So this is going to be a neural network.
And we replace this police officer by a discriminator, which also is a neural network, specifically
a binary classifier, which takes in some inputs and says whether it is real or not.
Whereas our generator here takes in some random inputs and then learns to output.
This bank notes such that the discriminator thinks that it is real.
Now we shall head on to the GAN lab, which is a project by Mensa, Kang et al.
And we'll consider some very simple example.
So, yeah, we suppose that the distribution is this year.
See that?
And then notice how we have a generator.
We have a discriminator, and then the generator takes in some random noise and then outputs this sample,
and then the discriminator takes in the sample and says whether it is real or it's fake.
Now, apart from this fake samples, it also takes in the real samples and then also says whether it's
real or fake.
Now, now the weight of the generator and the discriminator are being updated such that after some time
this samples which are going to be produced will look very much like this right here.
So let's go ahead and click on Run and we'll see what we get.
You see how we start with the training?
Let's take all this off.
See, we start with the training for now.
The fix.
Let's.
Let's pause and start over.
Let's restart that.
Okay.
You see, initially we get in these kinds of outputs.
You see this output.
You can look at this here.
So this is this is a kind of sample generated and this is the data of our real data right here.
And one thing we notice is that at times, as time goes on, you have your both the green and the purples,
which are considered to be real and then yours, all of the purples consider to be fake.
So this discriminator now starts making errors when it comes to seeing whether a given sample is real
or not.
Whereas on the other hand, the generator is now producing samples which look much like the real samples.
And it's because of this competition between the generator and the discriminator that is actually called
the Gans generative, adversarial.
This adversary comes from this competition between the generator and the discriminator.
And it now leads to the generator producing samples, which look very much like the real samples.
One other point.
You should notice that as we carried out this training, overall, we have two main parts.
We have this block right here.
Oh, let's take this block.
We have this block.
Which consists of wearing when the discriminator takes in this reel and then outputs some value.
The output from this is used to update the parameters of the discriminator.
And then when the discriminator takes this, and then when the discriminator takes in this, fake samples
this.
Output your is used now to update the generator.
So we have this block wherein we update our discriminator, we have the block wherein we update our
generator.
So with that now we could pause this and you could change the data distribution.
And so again, in comparison to with a VA is where we had this encoder and then the decoder where if
we have a distribution like this one, so we have some, uh, some input we, we want to have or we
want to be able to output or get outputs which will look similar to this input distribution.
And then after training this encoder decoder, we now break this up and then make use of only this decoder
to now generate output which are similar.
Ours distribution is similar to that of this real inputs right here.
Again, another important point to note is the fact that after training, after a certain number of
epochs at some point where when the reels and the fakes look very similar, the discriminator now becomes
somehow confused, as if you notice here you find some green patches.
You see, you have some green patches, purple patches.
Here we have some green patches and purple patches.
So it's no longer able to distinguish between the reals and the fakes.
And so instead of as before or as at the beginning, where I was able to say this is a one and this
was a zero, now it sees this as a 0.5 and this as a 0.5.
It becomes confused.
And so, in fact, the aim of our gang training process will be to.
Ensure that the generator wins the fight.
It should be noted that most very cool applications of gangs are in the domain of image generation.
And so we'll look at some of these applications in this article by Jonathan Hoy.
Gans can be used in creating anime characters.
You see here we have this anime characters which have been generated automatically using Gans.
And so this means that similar to what we had here, we are going to have a real data set which produces
images similar to what we have right here.
Let's get back here similar to what we have right here.
And then we're going to have a generator, which we'll learn over time to generate images which will
look like the real images.
And that's how we get to have images like this one.
Now, from here, we also have post guided person image generation.
Here, for example, you'll notice that we have this input image and then if we want to have this same
person but with a different post, then we could specify, we could pass in this post and then get this
output right here.
So you see that we have this input, does this image with this different pose and then this is being
generated.
Another application is in cross domain translations.
So you see here we have this input where we have this to say three zebras and then we are able to transform
this input image automatically into this other domain where we instead have horses.
And so this image or this input or this output rather, has been generated from this input and this
could be done in the reverse direction as we would see here.
We see you could get from zebra to horse as we had here, and then from horse to zebra.
Then we have another example.
This is a star gun which permits us to carry out translations or transformations where we modify specific
high level features of an image.
So right here, you see we have this input and then we add, for example, blonde hair.
So it changes the gender modified so that this this male becomes a female aged.
He's aged and talking about aged, you could build an application which tells you or which shows you
what you would look like after, say, 20, 50 or years.
Then here we have pale skin.
Another or some other modifications we could have.
Here is the angry, happy, fearful.
And so that's it.
You see, making use of this star again.
You could carry out these kinds of transformations on an input image.
The next we have this pixel again, which creates clothing, images and styles from an input image.
So you see we have the source image and we have this different images which have been generated from
this source.
We have other examples right here you could see, and then we have super resolution now for super resolution.
Gans have been used to increase the resolution of an input image while making this higher resolution
images more realistic.
So you see this image, for example, this by Kubik, this one right here, which has been gotten using
the by cubic method.
And then we have this other image which is using a rest net and then now you'll notice that there is
some difference with the kind of outputs we get using.
Again, notice how this image here looks more realistic or looks much more like the original as compared
to what this other two non gan methods produced to even get a much clearer difference.
Notice this part here where we have this water boring.
You'll notice how this looks much more realistic as compared to using a classical neural network like
the rest net.
Then from your move on to the next application, which is that of generating faces by this time around
very high definition faces.
So you see you have 1024 by 10,024 images generated you see this images of.
People would not actually exist, which haven't gotten using a pro game.
That's a progressive game.
Now we move to the next.
We have style again, which even comes with much better resolution and with some styling.
So from here we go on to high resolution image synthesis, where we could get you see the semantic map.
And then from here we generate this output we have right here.
Then the next Gauguin's, which as we've seen already, takes these kinds of semantic maps and thence
produces this output.
You see, this is the ground truth.
This is the exact output and this is what the GAN produces.
So that see from here we able to produce this.
From this we are able to produce this.
Now, this kind of technology could be applied in video compression in the sense that during a video
call where we have this input, so we suppose now we have this, this is a sender and then this is a
receiver we separated by this dotted lines right here.
So we have this input right here and then we carry out key point extraction where we get this key points,
as you could see here.
And then this is what's been transmitted via the network.
So instead of transmitting this input, we transmit this key points and then making use of some key
frame which has been passed.
Initially we combine those key points with the keyframe to produce.
Now this output right here, which looks like this original input which we wanted to pass.
And so now at the receiving end we are able to get this year at a much lower bandwidth since we are
taking in only the key points and not this whole input image.
Then we also have applications in text to image where we could pass in a text like this flower has long
ten yellow petals and a lot of yellow enters in the center and this generates this kind of output.
Now, talking about those kinds of applications, we could check out Korean dot com, which is in fact
a dial E mini model where we'll be able to create much more realistic output.
So let's click on this and while this is loading, we could check out on the other applications on the
next application text to image.
We've seen this already face synthesis.
You see right here we get with a single input image.
We create faces in different viewing angles.
So for example, we can use this to transform images that will be easier for face recognition.
So if we suppose that we get in this kind of input, let's say we get this kind of input we could generate
or we could convert this, for example, into this one year such that it will be easier for a face recognition
model to do its job.
Now from here we go to the next one image in painting.
Okay, so right here we have this input.
Let's increase this.
Um, take this off.
So as we're saying, we have let's, let's pick this one.
For example, we have this input right here, but we have this patch which has been taken off and now
making use of again, we are able to generate an output which will be like this input without the patch.
So just like again takes this patch off and as you could see, the again does this job quite well.
Let's reduce this.
There we go.
Before we move on, let's get back to this output created our icon dot com and you can see that this
doll E mini model produces even much more realistic images.
Then we move on to the discourse, Gans where we could create outputs which match the style of a given
input.
So supposing you want to go out on a little trip, you want to say take this back and you wanted to
get some ideas of the kind of show you could put on.
Then you could make a call on the this score again and you'll get this kind of output based on your
input.
So this is a model which is similar to the cycle again, which we are already seeing right here, and
where we were able to leave from one.
Domain to another.
One other fun project will be to generate emojis from input images.
So here we have this input image, for example, and we have this emoji which is generated from the
input.
Then another very interesting application will be in the blurring.
So right here we suppose that we have this input image which is clearly blurred and then we want to
blur this image.
You see that we're able to produce this kind of images making use of Gans, and you can see from this
images here that again, it's do this job quite well.
Another application is in photo editing.
And so now you do not need to be some X part in photo editing to carry out some of this photo edits.
All you need now is some gain and you're good to go.
Apart from image Generation Gans tool can be used in music generation though in this course we shall
focus on image generation.
Then the medical domain Gans could be used in anomaly detection, and that's it for this section.
In the next section we are going to look at how this Gans are actually trained in practice and the type
of loss functions we use when training this Gans.