Hello, everyone, and welcome to this new section in which we are going to be building our own variational
autoencoder models from scratch.
Previously we saw how variational autoencoders could be used in helping to generate new images where
we build out this encoder decoder structure such that we could produce outputs which are similar to
the inputs while being entirely new images.
In this section, what we'll be doing will be to build our own variational auto encoders and generate
our own images.
The data will be used in training.
Our variational auto encoder will be the MNIST dataset which you could get from TensorFlow datasets.
So right here we load this dataset and then we concatenate both the training and the test datasets.
So generally we usually have a dataset made of say X train and y train X test y test.
But since here we are not going to be making use of this outputs.
That's the Y train and y test.
We just get this tool and then we concatenate both since we we're not be having a test set.
So basically we have that.
And then one other modification we make is we are one of the preprocessing step we take is we divide
this values by 255, so we normalize our dataset.
So let's run this and then once our dataset has been downloaded, what we do now is we convert this
dataset into the TensorFlow data format.
So we have our dataset, which is TF dataset or rather data TF data dataset from, from tensor slices.
So you see that we take that and then we pass in our MNIST digits, which we've already downloaded.
So we run that and that should be fine.
Now we could check out the length of this dataset and you see it should have 70,000.
So we have 70,000 different data points which make up our dataset.
From here we're going to define the batch size.
So we will have a batch size of 128.
That's it.
And then we'll go to the usual steps of shuffling our dataset.
We have our dataset, we shuffle, we batch, and then we prefetch.
Now if you're new to this, you could check out the previous sections in this course.
Anyways, we have this three as we've said, and then now we could run this.
So that's it.
Could say train.
There is sent.
There we go.
You see that?
We have this train data set here, and we we could see its shape.
So it's all 28 by 28 by one images we have in our data set and there are 70,000 of them.
So that's fine.
Now, getting to the modeling, we're going to start with the encoder.
You can recall that what we had seen so far was, oops.
What we had seen so far was this model or this encoder model, which takes in an input image right here
and then outputs the mean and the variance.
So we have the mean and the variance and then this tool have been, um.
Combined via the reparameterization technique where we have mu plus sigma.
Times a random value drawn from a normal distribution.
And then this Z is passed into a decoder here.
So we pass this into a decoder and then we get an output image such that the the difference between
these two is minimized.
Now, that said, let's get back to the code and we design our encoder.
So our encoder here is going to be a very simple convnet.
We'll start by defining the latent dimension.
Um, there we go.
Let's just put this right here.
So latent dimension will be two.
We have that, and then we get back here for our encoder.
We're going to start, as we said, with this encoder input.
So we have an encoder input.
And then this this has a shape which we're going to give to be 28 by 28 by one, just as we just just
the same as that of our images in our dataset.
So that's the input.
We've seen this already.
And then from here we'll define a conv 2d, a conv 2D, which um, has 32 filters, uh, three by three
activation.
RELU So we're supposing that you already have some background knowledge and convnets activation?
RELU Uh, number of strides equal to the padding.
Same.
So we're going to build this very basic, um, convnet.
Now, this takes in the encoder inputs.
Remember or recall, we're using the the Keras functional API right here.
So we have encoder inputs.
We then create another conv layer.
We'll just basically copy this and paste it out.
And then what we'll have here is an increased number of channels.
So we have 64 here and from here we'll go ahead with flatten our outputs.
Now note that here we have X, so we should change this to X From here we move on to flatten so or we
have flatten and this takes in x.
So now the output is flattened.
We are now going to output both the mean and the standard deviation which we are going to use in sampling.
But before that we'll pass this into another dense layer.
So here we have this dense layer, say 16 outputs, uh, activation, activation, RELU And then we
take in X, Okay, so we have that and now we're ready to let's just copy from here.
We're ready to, to get the mean and the standard deviation.
Copy that.
Paste it out here and this.
Okay, so here we have the mean and we'll have the standard deviation.
So we have dense activation.
RELU Standard deviation.
And then here, since remember we have uh, this output to be the latent dimension.
So here we have instead of 16, we now have latent dimension which we've already fixed right here to
be two.
Now, one very important reason why we, we have this activations to be relu here will be simply the
fact that the the mean and the standard deviation are all positive numbers.
So because, uh, we may output or we may get negative numbers here, we want to always make sure that
the values we get are positive.
Now.
The problem with a standard deviation, particularly, is the fact that it's usually a very small number
between 0 and 1, where the number is very far away from one, meaning that the number is very instead
closer to zero.
So we'll have a number very close to zero like this.
But then the problem with working with the RELU is that having to find derivatives around this, uh,
zero here will lead to numerical instability during training.
And so what we want to do instead is to map this range of values or this possible range of values that
the standard deviation can take to a larger range.
Now, to carry out this mapping, we have to use a function.
Which is both continuous in this range and monotonous that is either increasing or decreasing.
Now, one great function for this task will be the log function.
As this log function will map values of X, the range 0 to 1 to values in the range.
Oh, let's open it up here.
And the range negative infinity log of the limit as we go to a zero of the log is negative infinity.
If you plot out the log, you will have something like this.
Um, let's have something like this.
So you find that as you go towards zero log that goes towards negative infinity.
So, uh, that's why we have negative infinity here and the log of one.
Is zero.
So we go from this range to this larger range.
Hence we can have a much more stable training process.
And so what we'll do now is instead of relying on this, uh, RELU activation to ensure that our standard
deviation is always positive, what we'll do is we'll instead compute the log of the standard deviation
square, which happens to be the log of the variance.
So we're going to take this off for both the standard deviation and the mean.
Let's take that off.
And then right here we have, um, log var.
That's a log of the standard deviation square, uh, log var, and then we'll move on to the sampling
process where we're going to obtain Z.
Remember, Z is equal MU, which is the mean plus, uh, the standard deviation Times Epsilon where
epsilon here is a random number drawn from a normal distribution.
We are now going to create the sampling layer which takes in the mean and the log of the variance and
then outputs the Z.
So here we have Z equals sampling.
There we go.
We now taking as inputs the mean and the log of the variance and that's it.
So from now we're going to create the sampling uh, layer.
So let's go ahead and create the sampling layer.
So we have here sampling, sampling.
There we go.
And this is a layer.
Okay, so we have that.
Then we just, we just have this call method which takes in our inputs.
Let's, let's just have our inputs.
And then from those inputs, what we'll have is the mean and the variance.
So basically we have what we extract, the mean and the variance from this inputs.
Remember, we have the mean and a log let's, let's say mean and the log var actually.
So we have that mean and log var which we extract from the inputs and then.
If you remember what we have here to the way we obtain Z is mu plus sigma times this random number right
here.
Now let's let's see how when given mu when we, when we get mu and we get the log of the variance,
we are able to obtain this sigma because we already have mu.
But now we need to get sigma.
Now note that sigma is a standard deviation and the standard deviation.
Or let's say let's just write Sigma and sigma itself.
Is equal, the square root of the variance.
You see that it's equal to square root of the variance.
So it means that, um, and then we know that the variance written like this can be written as e plus
exponential to the power of log of the variance.
So generally we know that X equal E to the log of x, you see that.
So here we have E to the log of the variance and then we also have the square root.
So let's have this here.
Now this is equal E to the log of the variance.
Let's just leave that as V and all of this to the power of a half.
We also know that e to the power of E to the power of X, to the power of all of this, to the power
of E is equal.
E to the power of a x.
See that?
So here this is going to be equal E to the power of half times log V, so it's going to be half log
variance.
So now, now that we have the log of the variance to obtain the sigma, this basically what we need
to do.
So let's get back and then we have um, the mean plus the sigma, which is a exponential.
And then we have a half, so 0.5 times the log of the variance.
And then we need to multiply this year the sigma by our um, random value.
So we have the random normal and then we specify a shape which is simply the batch size and the latent
latent dimension.
Now that we have this, we can now go ahead and define our encoder model.
Call this encoder model, which is uh, TensorFlow model, and we have encoder inputs for the input.
There we go.
That's it here.
And then for the outputs, we have this list made of, uh, Z, the mean and the log variance.
So we have mean and log var.
Okay, so let's give it a name.
We'll call this encoder.
Now, once we have this, we could, uh, get a summary of this.
So we have encoder model.
Summary.
Wish we could visualize right here.
See that?
From here.
Now we will move on to defining our decoder.
The decoder, as we've seen already, will take in the Z here.
And then I'll put the images.
So we'll go ahead here and create the inputs called as latent inputs.
And then there we have our TensorFlow input and shape is going to be the same as that of Z.
So right here we have latent Z and that's fine.
Okay, so now we have this.
We're going to take our Z, Let's get back here.
So now we have to upsample this year, which has a shape to now an image of shape, 28 by 28 by one.
But generally what we've been doing is we've been used to Downsampling.
So in Downsampling or we have an input, we have some convnet layers, we stack them up and then when
we get towards the end we could flatten and then we have some dense layers with a specific output which
matches the type of output we want to get.
But in our case, we're now doing some sort of the opposite of this.
So what we'll do is we'll go through some dense layer here, we'll go to some dense layer, and then
from this dense layer we would pass this into a transpose convolution layer.
So we've been using the convolution layer, but here we'll just use the Conv 2D transpose layer.
Essentially what we have is this come to the layer with its weights which upsamples inputs.
Now getting back to the code, we have this input which has shape batch size by letting dimension let's
say two.
And then from here we want to make use of a conv 2d transpose layer which takes inputs batch size by
some x, by some y by some z.
That's similar to the conv layer.
So what we will have to do is we're going to reshape this actually.
So we have to reshape this such that we have something like this.
Now if we want to have a x, y, z such that X is say for example, seven, y is seven and Z is say
64, Let's take the example.
Then we have to ensure that what we getting after this has shape be by um, this seven by seven by 64,
seven times seven times 64.
You see, um, getting back here we have this.
Let's, let's, let's maybe redraw this again so it's clear.
So what we're saying is we have this input and then what we intend to have is something like a batch
size by seven, by seven by 64.
Now, the reason why we're picking seven is because the output is 28 by 28.
So one, to be able to upsample sorry that we could say seven by seven Upsampled to 14 by 14 and then
the 14 by 14 Upsample to 28 by 28.
So that's why we're picking seven here.
Now, uh, this here, you see, it doesn't match with this.
There's no way we could reshape to, to become this.
So what we'll do is that we'll pass this through a dense layer which has outputs batch by seven times,
seven times 64.
And now after reshaping, we could obtain this.
So let's get into the code and what we'll have is our dense layer, dense, oops, we have our dense
layer, um, seven times, seven times 64 and then.
We have the activation.
Activation.
RELU And this simply takes in the latent inputs.
So we have here.
Latent inputs.
There we go then from from here.
Now we do the reshaping.
So we have x equal reshape and then we specify the shape.
So we have seven by seven by 64.
So that's it.
Now we have that.
We have X, there we go.
We now start with our conv 2D transpose.
So you see we reshape this into this.
Now could make use of our conv to the transpose come to the transpose which takes in a number of filters
is very similar to the conv layer.
So we have a number of filters, let's say 64 filters and then the kernel size three, the activation.
RELU So we're going to use the RELU activation number of strides.
Um, two the padding is going to be the same.
See?
So it's quite similar to the conv layer, but with the difference is that now we upsampling instead
of downsampling.
So we have that.
Then from here we are going to change this to 32.
So for the encoder, what we did was we increased this year this number of channels and then here we
reduce the number of channels.
Now in our final output layer, we're going to have this decoder output, which is going to have just
one channel.
So here we have an output which is 28 by 28 by one where the values lie between 0 and 1.
So what we're going to do now is we're going to have channel numbers equal one.
Um, the activation instead of RELU will be sigmoid.
So here we have sigmoid and then we're not going to use any strides since we're not Upsampling So that's
it.
Now, the reason why we're using sigmoid here is quite simple since one of our values will be fall between
0 and 1, we want that each and every time we have an input, we have something like this.
So no matter the input we have, the sigmoid will always put the the value or make that input turn into
a value which lies between 0 and 1.
And that's basically what we want to do in this last layer right here.
So that's why you see we're making use of the sigmoid.
Now once that's done, we create our decoder model, which is TensorFlow model, and then we have our
latent inputs.
There we go.
We have our decoder output.
Uh, the name is Decoder.
And then we could, uh, get a summary of this model.
So that's basically it.
Let's run that.
And there we go.
See, we have our decoder model now for the training.
We are going to make use of the Adam Optimizer with the learning rate of of 0.001 and we're going to
train for over 20 epochs.
Now, as we've seen already, our loss we made of two parts is the reconstruction and the regularization
part.
Now for the reconstruction part, our aim is to minimize the difference between those output at image
and the input image.
So we'll go ahead and start with the reconstruction loss.
We have our custom loss.
There we go.
It takes in y pred Dixon y.
Let's start with y.
True.
Dixon y true.
Y pred.
And then the reconstruction loss itself is defined such that we have recons.
Let's just say loss.
Reconstruction is equal.
Tf.keras losses are binary.
Cross entropy loss.
Okay, so our outputs remember, our outputs range between 0 and 1.
So we could use the binary cross entropy loss here.
Feel free to test out our different losses.
So we have that and then we pass in y.
True and y pred.
Now, once we make use of this, we are going to now sum all the values because we haven't.
Let's say we have something like this.
Let's suppose that it was a five by five output we were having.
So we'd have something like this.
One, two, three, four, five.
That's fine.
Five.
There we go.
You see, we have this five by five here.
So with this binary cross entropy, we'll be able to get the the difference for each and every position
here.
Now, we need to sum all this here.
So what we'll have is, um.
Let's take this off.
We'll take this off and we'll have the reduce some.
So this reduce some now we'll some all this different different all this different loss values we get
here.
So now we we some for each and every position obviously we have 28 so 28 by 28 positions.
We also need to specify the axis here.
So the axis we're going to work with is one and two, one and two.
Now to understand why we have having this, let's take a look at the shape of the output is B by 28,
by 28 by one.
But where we actually carrying out this, where we actually computing the loss is in this axis here.
So we specify one two because this is zero one, two, three.
So this is where we want to, um, compute our loss.
So it's on this two axis now said we have that and then we now look for the mean so we could after summing
up, we could look for the mean that's average the values and that should be it.
So that's it for our um loss reconstruction.
The next step.
Let's take all this off.
The next step will be this last regularization getting back here.
You see, we have this sum.
See the sum of, um, the variance plus the mean square of the mean minus one, minus the log of the
variance.
So let's get back here.
And we have this negative half here.
If we take this negative and multiply by each and every one of this, we'll have um, log var plus one
minus the mean square, minus the var.
Um, before we continue, remember again that we could get Sigma J as E to the log of, uh, Sigma J
See that?
So also, Yeah, yeah.
No, let's, let's, let's, let's rewrite this better.
So we have sigma J equal E to the log of Sigma J.
Okay, so that's it.
Um, let's get back here.
We have our loss.
Uh, we'll still have this mean and sum, so we'll say of average sum and then find the average.
Uh, let's have that and then we'll have.
-0.5
times the log var.
Let's get here.
We'll need a log.
Var.
Oh, let's say me mean and then let's let's let's get exactly what we had here.
Remember in this, um, encoder model, we outputted the mean and the log var.
So let's have mean.
There we go.
We have mean and log var.
Okay, so we have this set.
Now, we could make use of it right here.
We have, as we've said already, log var plus one.
So we just have plus one minus the mean square.
So TF math mean or rather the square of the mean and then minus tf math E to the power of log var.
See that?
And now we return the last reconstruction and the last regularization.
So that's it.
So this is our custom loss.
We run this, we're getting this error.
Let's add this and run that again.
Okay, So that's fine.
And we now set to start with the training, but before going on, we have to um, also specify the axis
for the sum.
So right here we have this axis which is equal one and we explain why we, we specify this axis to be
one.
Now the shape of the log var and the mean is this batch by two.
And so the shape of all the sum will still be this.
And so if you're comparing this, all right, if you're competing this year, you'll get this kind of
output.
And so since for the loss, we need a single value.
We need to sum all the, um, values we get in this axis.
So that's why you see, we specify the axis to be one.
We have this, uh, input which we define.
Then we have the encoder model which outputs Z, the mean and the variance, which we're not going to
make use of.
And then we have the decoder which takes in Z and then produces this output.
So here we have this uh, model which contains both the encoder and the decoder.
Now we're going to go ahead and build our custom training block.
We suppose that you already have an idea of how this works.
So we have that training block which takes in our X batch, and then we're going to make use of tensorflow's
gradient tape.
So we will have this with TF gradient tape.
Gradient tape.
There we go.
As a recorder.
We're going to pass this batch into the encoder.
So we have our encoder model, which takes in X batch.
And then what it outputs is Z, the mean and the log variance.
Here we have this.
Okay?
So we have that set.
And then now once we get this, we, we, we, we get the Z from here and pass this into our decoder.
So we have our decoder model which takes in Z and then what it outputs is our Y predicted.
See that?
And now from here we could obtain our loss by simply calling on our custom loss method, which we've
defined.
It takes in the Y.
True.
It takes in the y pred widespread.
It takes in um as we've defined here, the mean and it takes in the log variance.
Now this y true.
Y true.
Happens to be the batch.
Remember we are having this year.
This encoder takes in that.
So we have our input image here.
Let's let's take this off and draw it a bit here, a bit clearer.
So here we have this and we have this.
So we have our encoder and we have our decoder, we have our input image, which is what we expect to
have here.
So our white arrow is what we pass as input here, which is this X batch.
So that's why you see we specify Y trust x batch, then Y pred is what the decoder produces.
So we're going to compare the y pred and then the Y.
True, which is, as we've said already, the X batch.
Um, that's fine.
We get back to the code, we have that set and then now we have our partial derivatives or partial derivatives
which will get by making use of the recorder.
So we have recorder gradient.
It takes in the loss, it takes in the overall model's trainable weights.
So we have trainable weights.
Okay.
So talking about the overall model, which we yet to define, we have it right here.
It takes in this input 28 by 28 by one.
It takes the input of pass into the encoder model, gets the output from the encoder model and then
into the decoder model.
And then from there we create our model, which takes in the input and this output right here.
So from here, let's get the summary and we see that we have exactly what we expect.
So we have this model which takes, as we've said already, this input, the encoder outputs, Z, the
mean and log VAR, and then the decoder outputs this, this image right here.
So let's get back to our training as we're saying, where we have this, uh, partial derivatives from
the loss and the trainable weights.
So that's it.
We then go ahead with the gradient descent step, we apply the optimizer, so we have optimizer dot
apply gradients.
There we go, which takes in our partial derivatives and our trainable weights.
So there we go.
We have the partial derivatives, which we've just calculated right here, derivatives.
And then the trainable weights, trainable weights.
There we go.
Okay, so that's it.
We could from here just simply return the loss.
Now we're going to run this and let's we haven't run this here.
Okay?
So we have that already.
Now we could define our neural learn method.
So we have here neural learn which will take in the number of epochs.
And then from here, for the epoch in range epochs.
There we go.
We're going to start by printing out the training starts for epoch number, whatever epoch we are at.
So we format that to take in the epoch plus one since we're going to start from zero.
Or we could just take this one from here and then there we go.
Now we have to add plus one here.
Okay.
So we have that.
And then now we're going to do for step um, X batch, we're going to enumerate in enumerate, um,
the train data set, which we've defined already.
Let's get back to the top and we see we have our train data set, so we're going to go through this
train data set.
Let's get back here.
There we go.
So we go through our train data set, we take a specific batch and then we compute the loss.
So we have that training block, which we've defined here.
Our training block has been defined here.
So not only we compute the loss, but we also apply the gradient descent step for that specific batch.
So we're doing this for each and every batch of our training data set.
Now, what this takes in is our batch, simple as that.
Now once we have this, the next thing we'll do is we'll print out our loss.
So the training loss, um, is, um, there we go.
We have loss.
And then once the training is complete, we could simply print out.
Training complete.
Okay, so we have that set.
Now let's run this and then we have neural learn.
Let's train for epochs.
Um, yeah, we're getting this error.
Let's take that off.
Let's.
Fine, let's run this again.
So training is now complete, and here's what we get.
You see that loss drops and then we can now get straight into testing out our model.
Now, before testing, let's recall that our VA is comprised of two units.
That's our encoder.
And then the decoder right here.
Now we've trained this model end to end to make sure that the inputs look very similar to this output
produced.
Now, if we want to generate a new outputs, what we'll do is we are going to cut off or we are going
to take off this region here and focus only on the decoder now to generate an image at random or to
generate a digit, in our case, a random.
We'll just have to pass in a Z in here.
That's a random value of Z.
Remember, Z is mu plus sigma.
Epsilon.
So we have to pass this value of Z in here and then get a random output.
Now, remember, Z is two dimensional.
So Z is this vector made of two values.
And so that said, we will define the first value here, which we will call grid X, and we use the
Linspace method to get values from a scale.
Let's add a cell above this so we will have a scale, um, which will take a value of one and then we
will have n equals to say 16 different values.
So here we go from -1 to 1 and then we'll have 16 values in between.
Uh, repeat this for grid y now.
So this is the first, uh, element and this is the next element.
So here we will have different elements so that we could generate different images.
There we go.
Let's run this and then print out our grid X and our grid y.
Now that we have this done, the next thing we'll do is we'll plot out our different images, which
you shall generate using our decoder.
So here we have this figure.
We define the fixed size, let's say five and we five by five.
And then.
For I in greed X.
And for J in greed.
Why?
We define the different subplots subplot.
Um.
Five by five by K plus one.
Let's define k right here.
So K equals zero.
Okay, so that's it.
This is greed, not greed.
We have greed that way.
And then now we're ready to use or to use this greed X values.
That's the values of I and J to generate new images.
Okay, so what we'll do now is this is plt dot subplot.
So what we'll do now is we'll have our input which is TensorFlow constant and then we have J.
So these two values.
And then from here we have the output which is our V, but notice how we pick out, uh, layers too.
So, uh, to better understand this, you should get back here where we defined our model.
So here you see, this is, uh, the variational auto encoder, which is made of different layers.
So, uh, this is layer zero, layer one, and then layer two.
Um, if you do v dot layers, um, let's say zero.
See you have this input layer right here.
Now, if you change this and say, anyway, let's let's say for I in, uh, range three.
There we go.
We're going to print this out.
So we're going to print our layers.
I.
Let's run that.
And we see that we have this input layer, we have this functional model, this other functional model.
But you should note that's basically our encoder and decoder.
Okay.
So that's set.
We're going to make use of our layers here.
So we have the layers to this is to say that we're using the decoder actually.
So our decoder is going to take in the input, see that it takes in the inputs.
There we go.
And then we'll have.
To select our first axis here.
So that's why we have this.
And then we have to, um, select this other axis right here, so such that our output now, you see,
if we, if we do this, if we have zero here, we do this.
See, this is not going to be transformed to 28 by 28.
So basically that's why we do this.
Now once we have that, we do the show.
And then we have our output.
Our map is, uh, gray.
There we go.
Plot axis.
Um.
Of.
And then K plus equal one.
Okay.
So we're basically increasing the value of one of K, sorry, such that we could have this as different
subplots.
So let's run this now and then see what we get.
You getting this error here?
So this is this is actually because we're having many more values as compared to what we define here
in this subplots.
So what we should have here should be n and here should be n that set.
Let's run this again and see what we get.
Now, as you could see, we are able to generate this digits, making use of just this Z vector right
here, which is composed of two numbers.
Now, one thing you can notice here in this latent space is that as we go from values of -1 to 1 in
this two dimensional latent space, the outputs are created or generated such that in each line, as
you could see, we have one digit, which is being.
Slowly morphed into another.
You see here we start with a nine.
But as we change values in this latent space, you see how slowly we get to eight.
And then here you see eight.
And at this point you start with a nine and then you slowly get into fives and so on and so forth.
You can look at this horizontally as well as vertically, see that we get to six and that.
And then you could also look at this diagonal, you see nine, eight, three, two, six, and then
you get two zeros.
So, um.
For this first line or this first lines.
You can look at this first lines as going from 9 to 8.
Then here is like from 9 to 5, but passing through three C and so on and so forth.
So basically this is how we generate images using variational autoencoders.