Hi guys, and welcome to this session in which we are going to write the code for generating images
like this in TensorFlow.
From the previous session we had looked at the GAN loss function and now we will see how to adapt this
loss such that we could instead use the binary cross entropy loss.
And then from there we'll go ahead to look at different methods to make our GAN training much more stable.
You can see from here that this GAN training isn't a very stable process since, um, we have two adversaries
where one is trying to maximize the loss and the other is trying to minimize the loss.
So by nature, Gans are much more difficult to train as compared to other classical neural networks.
Then after looking at how or looking at different methods to make this GAN training process more stable,
we'll go ahead to train our GAN.
From our previous session, we had the discriminator loss and the generator loss.
Now we want to introduce the binary cross entropy loss.
You'll notice that it looks quite similar to what we have here.
Now let's start first with the discriminator for the discriminator.
It takes in this real image and then we expect it to output A1C that we expect to have a wander.
Now, uh, for the binary cross, if we have to compare this with binary cross entropy, then this will
shuffle this or wire is going to be equal uh, D of x I, which is practically what we have here.
So this is our Y Chapo And then, uh, yeah, y this y Chapo is going to be this here.
Now when, uh, since we want our Y to be equal one, it means that one minus Y will be zero.
So we will not consider this, but we'll consider only this part right here.
Now, if we consider only this part, we are left with this expression since Y is equal one.
And then when Y equals zero, the case where we have this sorry, this Y Chapo equals zero, then we'll
have log of zero, which is log of negative infinity minus that with this minus which we didn't have
here is now in this year because we're dealing with the binary cross entropy, the negative of the negative
infinity gives us positive infinity.
So it means that, uh, for y Chapo equals zero, we instead have positive infinity.
So here, since we want to have Y Chapo to be equal one, then, um, our aim will be to minimize the
binary cross entropy loss since the values we could get will range between zero and positive infinity.
Unlike here, where the values were ranging from negative infinity to zero.
So here was maximization problem.
We had, uh, gradient ascent.
But here we're trying to minimize this instead.
So we minimize this because we want to obtain a zero since it's when our output is zero.
That is when our Y Chapo is equal one that we get this output of zero.
So that said, we have it for this first part.
For the next part, when we're dealing with, uh, our data, our fake data, which has been generated
by our generator, we expect D to be equal to zero.
And since our Y in this case is equal to zero, since we have Y to be equal to zero, then this term
is taken off and we're left with only this term.
Now, for this term we would have one minus the expected value of D, which is zero.
That's one log, one minus y Chapo, where y Chapo is what the model predicts.
So, uh, in the case where we have this, y sharp ought to be equal or zero as expected, then we would
have log of zero and that will give us zero.
So we would have a zero here.
Now, in another case or in the other case where our model instead predicts one year.
If the model instead predicts one, then we'll have log of one minus one, which is going to be zero.
Log of zero is negative infinity.
This negative here turns out to positive infinity.
And so now we have positive infinity.
And since our aim is to obtain this Y Chapo to be zero, it means that we will minimize this expression
here.
Such that.
It takes the value of zero or with the with the aim of having a value of zero.
And again, unlike this original expression where we went from negative infinity to zero, now we're
going from zero to positive infinity.
And so this means that when working with the discriminator and making use of the binary cross entropy
would go through a simple gradient descent where we will minimizing the loss.
From here, we will move on to the generator.
For the generator.
Well, we expect the to produce a one right here, so we expect them to have one year.
And this means that this expression is taken off since we have one minus one, which is zero.
So here we are left only with this.
Now we're left with one is one.
So it's just log log of Y chapel, which is what the model will predict or what this the, the discriminator
would predict.
And so we have negative anyway.
Let's just say negative.
Let's take up the one on N and the sum.
So we have this expression right here.
And then in the case where the model actually predicts a one, so the model is expected to predict a
one and it actually predicts a one.
In that case, we would have a zero in the case where the model predicts a zero when when it's supposed
to predict a one.
In that case, we will have log of zero.
That's negative infinity.
Negative negative.
Infinity is positive infinity.
And again, here our aim is to obtain an, uh, this zero right here.
So we will again be minimizing our loss.
Now, given that we will be implementing the model or the architecture in this paper, that is the decision
paper.
We should note some of these guidelines here.
We should replace any polling layers with strided convolutions.
So instead of using pooling, we use strided convolutions and fractional strided convolutions for the
generator.
The next use batch norm in both the generator and the discriminator then remove fully connected hidden
layers for deeper architectures.
Basically here we're using the convolutional layers instead of the fully connected hidden layers.
Uh, use RELU activations in generator for all layers except for the output which uses the tank activation,
then use leaky RELU activation in the discriminator for all layers.
But before we move on to look at some of these details, we should note that there is this GitHub repo
here by one of the author authors of the paper that's um.
There we go.
Let's get up here by Sumit Chintala.
And what he proposes here is this, um, list of tips and tricks used to make Gans work.
Now, as we said already, it's not that evident to make Gans work.
So, uh, taking advantage of the experience from one of the authors of the original of the GAN paper,
not the original Gan paper will be very interesting for us since we will not, uh, get to make the
same mistakes which maybe he had made, uh, before discovering all those different tricks.
Now, that said, we have here the very first one is you should note that this list is no longer maintained,
and I'm not sure how relevant it is in 2020.
So this is this has been for a while.
Uh, let's see here.
Six years.
Anyways, we have, uh, normalized the first one, normalize the input.
So normalize the images between -1 and 1.
Then use Tench as a last layer of the generator.
Again, this was already in the paper.
The next tip will be to modify the loss function.
Now, it should be noted that we've already explained this, but maybe the transition from this previous
loss function to this other loss function wasn't made very clear.
Now let's get back to that call for our loss function.
We had, um, we had a log of one minus D of G of Z.
This is for the generator, remember?
So we had only this here, only this expression in the original GAN paper.
They expected the discriminator to produce a zero, although previously we had mentioned that we expect
the discriminator to produce a one right from the beginning.
We've been speaking of this, but in the original paper what they actually wanted was the discriminator
to produce a zero for the generator.
Before we go on to look at how to compute this loss, we are going to take into consideration this modification
on this original loss right here.
And this modification comes in because of the following problem.
When the training just starts, it's difficult for this discriminator to output a one unlike here where
it's easier for it to output a zero when it sees fake data.
Because remember, let's get back to this and let's restart here.
Let's restart this and let's pick another distribution or let's let's pick this one which is slightly
more complicated.
So when you just start with the training, the the, the generated outputs, you see here, the generated
outputs do not look very much like the real data.
See, this here is very different from this.
And so because of this great difference at the start of the training is difficult for us to make the
generator fool the discriminator to output a one right here.
And also because of the fact that classifying whether an input is real or not is easier than generating
new inputs.
The generator here will experience vanishing gradients.
And so instead of, as we've seen already, trying to minimize this, we can instead maximize the sum
of log D, G of Z.
So instead of log one minus G of Z, we're going to have log D, G of Z.
The reason why it's preferable for us to use this expression where we maximizing this here instead of
minimizing this other expression is simply because when we make use of this expression, we are being
more lenient on the generator at the beginning of the training especially so when we are minimizing
the log of one minus D, G of Z.
We had labels, or if we were to make use of the binary cross entropy.
We lost.
We would have levels which are equal zero.
And so if our level equals zero.
Obviously this expression is left out and we left only with this part.
Now we're left with y equals zero.
And then we have log of one minus y.
Just similar to what we have here.
But then the fact that at the very beginning we are expecting our discriminator to output y equals zero
when it sees fake data from the generator is a problem because it's a very easy task for the for the
discriminator, especially because right here or at the very beginning this year is obviously not going
to look like the real data.
And so the discriminator will find it very easy in, um, predicting that this is fake data and because
of this is the generators weights wouldn't be updated any further to ensure that we could produce even
a more realistic looking fake images.
And so what we do is we flip the labels and flipping the labels matches with this expression, that
is, let's, uh, take this off.
We are having now y equal one.
So instead of y equals zero, we now, uh, have y equal one.
So we expecting the discriminator to output one when it sees fake data from the generator.
Now, uh, doing this, we have y equal one, So obviously this is left out.
You see that this now matches with maximizing this expression.
And so when y equal one, now we are telling the discriminator to output a one year when it sees fake
data.
And now this will permit the generator to be able to update its weights and so make the training much
more stable as compared to when we're dealing with or working with labels y, which are equal zero,
You should also note that when we're talking about labels here, we're talking about this y eyes.
And when we're talking about what the model predicts, we're talking about this y hat or this Y Sharples.
Our next tip is one which we've discussed already.
So we told in the GAN paper the loss function to optimize is the minimization of log one minus D, but
in practice folks practically use max of log D.
This uh, what we are seeing already where we had the minimization of one minus log D, g of Z.
So we had um, log sorry, log of one minus D, G of Z right here.
So we minimize this expression.
And then now instead of this, we maximizing um, the log of D, G of Z.
And the reason why we prefer to work this way, as we've said already, is because if you are expecting
the discriminator to take in some fake data from your at the beginning and say that.
It is fake, then this is going to be a very easy task, especially at the beginning, since the fake
data here is going to look very much different from the real data.
And so because we our aim here is to make this discriminator output, a zero is going to be difficult
for the generator to update its parameters such that the discriminator can start getting fooled.
And so instead of this, as we've said already, we flip the labels and instead aim for the discriminator
to output once.
For the next tip we set.
Uh, to say don't sample from a uniform distribution.
So for the noise, uh, here for our generator noise, we're going to sample from a Gaussian distribution
or normal distribution.
Okay?
So from there we encouraged to use batch normalization, avoid sparse gradients.
And unlike in the paper, if we get back to the Dcgan paper and we check, let's get to this.
We check here we told use leaky relu activation in the discriminator for all layers and then use relu
activation in the generator for all layers.
But what this here is, uh, the leaky Relu is good in both the G and the D.
That's both the generator and the discriminator.
Then one point you should note is the fact that the stability of the GAN game suffers if you have space
sparse gradients.
Now, what do they mean by this?
Uh, if you have a RELU activation, if you have some input, all negative inputs are sent to zero and
all positives remain the same.
Then for the leaky relu, we have all negatives which will take up some value depending on what we pick
that value to be.
If the value is 0.2, for example, then an input of negative one will give us an output of -0.2.
Uh, and an input of negative two will give us, for example, uh, -0.4 an input of, of say whatever
value times, uh, this times uh, -0.2 to get the output.
Now that said, the this, the positive section remains the same.
But now when we talk about the sparse gradients here, it comes due to the fact that if we have a relu's
in our network, then we will tend to have many zeros and the zeros are will cause this sparsity in
the gradient.
And so because we do not want to have this and because we want to train the the Gans in a more stable
manner, then we'll make use of the leaky relu.
Then for down sampling, we're told to use average pooling or conv 2d and strides.
Then for up sampling use pixel shuffle conv transpose convolutional transpose 2d with strides.
Okay.
Then from here we're told to use soft and noisy labels.
So what does this mean?
This means that if we have a discriminator and let's say we're training our discriminator where we have
an input from a generator or we train our we update the parameters of the, of the generator, um,
where we pass in.
Our fake data from this output of the generator into our discriminator.
And the discriminator has to compare the output from the model.
Let's let's put the output from the model in this color.
Let's say the output from the model 0.4, for example.
So what we'll be comparing will be this correct label with the model's prediction.
You see that now what this here is applying label smoothing.
So instead of taking one, we could take a random value around one.
So instead of this, we could take, for example, say 1.2 or 0 point 8 or 0.9 or whatever value just
around one.
So instead of having hard labels, just like some strict labels, either we have zero or we have one
or um, take values around.
So around let's change the color.
We'll take values around zero and then values around one.
You see that?
So we use smooth labeling instead of some hard, uh, labeling.
Then we also told to make the label, make the labels the noisy, make the labels noisy for the discriminator.
That's occasionally flip the labels when training the discriminator.
Okay, so that's fine.
Now, next, uh, use this again when you can.
It works.
So if you, if you can use this again and the model stable use hybrid model.
Like, for example, the and again, now here we have been told that if we are to generate images,
we should not use the original Gans, that we should not use, um, the simple neural networks that
are fully connected neural networks.
We should go in for convolutional neural networks.
Okay.
The next is to use stability tricks from reinforcement learning.
Now we get to treat your reinforcement learning.
So we're going to skip out this now from here using Adam Optimizer Optim.
Adam Rose So the app, the Adam Optimizer rules and then track failures early.
So if you want to be able to train your Gans without maybe regretting at the end that you you haven't
or you can get the kind of results you expected, you should try to ensure that you make sure your GAN
isn't doing any one of this right here.
So if you're training and your and your loss, the loss of the discriminator goes to zero, then it's
failure mode because your discriminator is proven to be too good at doing its job.
And then if the norms of the gradients are over 100, things are screwing up.
When things are working, the loss has low variance and goes down over time versus having huge variance
and spiking.
So what they're saying here is if we have this, the loss for the discriminator, remember we have the
discriminator on the generator.
What we expect to have is something like this and should go down slowly over time instead of having
this kind of, uh, high variance and spiking.
Now, if the loss of the generator steadily decreases, then it's fooling the D that the discriminator
with garbage.
And so this means that we do not expect the generator to be so good that during training its loss just
drops, um, steadily.
And so, as we've said already, you should track all these failures early on.
Now the next we have don't balance loss via statistics unless you have a good reason to.
Now, uh, they say they've tried.
It's hard and they've tried it all.
Um, let's take this off.
So what you're saying is.
Don't try to balance the training of the generator and the discriminator based on some loss value.
So if you are to try this, you should have a principle approach to it rather than just intuition.
Now, if you have labels, use them now.
Talking about labels, this means that if you have, say, we have this our discriminator and then we
have our real data and then you have maybe some dataset of fake data.
Then you could train your discriminator like the usual classifier in supervised learning.
Then the next point is to add noise to the inputs and then decay over time.
From here we have this tips where they actually not sure.
So or we may just keep them actually.
Uh, this is for conditional.
Gans will not take this into consideration use dropout and in both train and test phase.
So provide noise in the form of dropout as this generally leads to better results.
Much thanks to the authors Smith, Emily Martin and Michal.
Then, apart from the vanishing gradient problem, another very common problem will be that of mode
collapse, where the generator produces output or produces the same outputs even after training for
several epochs.
And so now we're going to start with building all this again while taking into consideration the tips
and tricks which we've just seen.