Hi and welcome to Lesson six in the deep learning section here.
Hi and welcome to the six lesson of the deep learning section of this course.
We're going to start with introducing regularisation now onto the same network we've trained just now.
That was the fashion amnesty F.A. dataset.
Previously, we didn't train it without.
We trained it without any augmentation.
Now we're going.
Hi and welcome back.
So what we're going to do now is we're going to train the same fashion amnesty to set.
However, we're going to introduce regularization into this model and we'll assess its performance and
you'll see how regularization has improved our accuracy.
So let's firstly get started.
So in this lesson, what we're going to do?
We're going to introduce four types of regularization.
We're going to do L2 regularization data augmentation.
Remember, this is the image manipulation one where we keep every time we load data into our dataset,
we just add some random manipulations to it.
So that means it helps it generalize much better to unseen data.
Then we're going to just drop out and batch norm.
It's quite simple to introduce increase.
So let's firstly begin.
Let's look at our model and you'll see it takes a little while because we have to connect to our instance
on Google Cloud.
And do we go?
So it's going to be loading this dataset?
No, that finishes quite quickly.
Check to see if we're using a GPU, just to be sure.
And we are we using the Tesla P 100 GPU again, we inspect our data.
I'm going to go through this a bit quickly because you've seen this before, and I don't want to waste
your time and be redundant.
Similarly, we have seen this two class plotting examples.
Next, we're going to take a look at something that you haven't seen before, and that's the image data
generator function in.
So what is this image data generator function?
Well, we need it when we're doing the data augmentation part of it because we're loading data.
You mentioned this and we're loading the data in batches, but we want to apply these random data augmentations
to that batch.
The way we do that is we used image data generator that basically points to these images, applies to
manipulations and then returns the images.
Where does manipulations back to a training loop?
So that's what this function does here.
So well, I'll let you listen shortly.
I'm just talking about it because it's mentioned here, and I just think you would.
It would feel curious about it at this point.
But the rest of the code here is standard stuff you've seen before.
We're reshaping the training that is said the test dataset.
Then we're going to do the hot one hot, including the labels.
We're going to get dual image rows and columns and created with image input shape variable.
We're going to normalize the data as well.
Next, we're going to keep a track of the number of pixels classes as well as due to one hot including.
And next we build our model and notice this model looks a bit longer than our previous model.
What's different about it?
Well, you should have noticed noticed three main things.
Firstly, we now have a parameter defined above here called L2 rhumba L2 and L1 norm.
We're going to be using the L2 norm, so too do to do that in the conflict here.
When we do model that add remember, we specify no filters, the kernel sized activation, and there's
a parameter we can introduce called Colonel Regularised and we just use a regular dot l to remember
we import regular races here.
We could do regular as a dot L1 as well.
We give it the L2 variable setting.
We were using point zero zero one here in this instance.
You can try different values as you remember the bit larger to value, the more you penalized a weight,
so the slower.
Generally, this network would converge to.
So now you can see we do it here as well with a second cornflower.
And notice something he actually skipped over it accidentally, but we're introducing Batch Gnome.
And you can see how easy.
But still to introduce it simply just is just modeled on ad batch normalization layer, which we import
up here and you place it between the currently, as you can see after the real layer, which is what
happens in this little block of code here.
This is a conflict with the reload activation along with the L2 razor.
So then we just add a batch norm at the end right here.
No, it doesn't.
Just one thing to note that all the layers that have the L2 that can take the kernel regular are there.
Those layers need to have the same URL to regularize a set as well.
I mean, yes, you probably can mix L1 L2, but it's not recommended unless you know what you're doing.
So generally stick to the same regularised for each layer.
So that's applied in the dense.
And the conflict is I mean, do you know why?
Because those are the layers here you can see besides about stone, but the conflict is here and the
dense layers away, all of the learnable parameters are the bulk of them.
So that's those are the weights we want to control.
So you can see we have the batch gnome here.
We have another batch gnome here.
And next we're introducing Drop-Out.
We're introducing drop out after the Max Boulia.
In this case, we can introduce dropout at different points, but it generally works best after the
max boot or after completely a conflict output.
So we're setting the dropout rate at zero point two.
Generally, I like to use values of point to point trips to find those tend to work well.
I don't go up to point five because that slows down convergence time quite a bit.
So point two is a fairly good one to go at.
So you can see already we've introduced one to three types of regularization almost effortlessly.
You can see how easy it is to configure CNN's now.
And next, we must do more to compile with a loss, optimize a specified A metrics print or model summary.
And we get that below.
So now let's begin trading on model.
However, let's note something in this code because this code does look a bit longer than our previous
training block of code that generally just had a model that fit and specified a bad size and epochs.
Well, what we've added here is the image data generated that I mentioned previously.
You can see this is creating a generator here with the specific augmentations that we want to use.
So we re scaling it where this is normalizing it.
Basically, this is rotations which shift height shift share in the Zoom range horizontal flip.
All of these should be self-explanatory.
However, if you want to take a look at what they actually do, go to the decrease augmentation documentation.
I'll probably put a link within this notebook if you want to see afterwards.
It's not in my video notebook, but I'll post one in here for you guys so you can take a look and it
will basically show you what these things look, look, look like.
How you can like this full mode of what the film would nearest means.
Basically, what narrowest means is that if you were to zoom out or shift an image left or right, you
would end up with some black pixels or some and some blank pixels, basically.
So you can fill that with either the nearest pixel.
So whatever pixel was nearest, it has copies that value.
It's quite good to you as near as I would think.
So that's we have a image data generator I turn to here.
So now when we're doing model that fit, we use that generator we created up here and do dot flow.
So because this is this, this is just how Keros treats its image generator.
So we used this as a function built into it.
So it's trained in this agenda of.
Variable for it to flow.
And then we specifies our training data or training labels, the bad size, and then again, we just
do the epochs validation steps for epoch.
Now notice this parameter.
I don't think this was here before in the previous lesson.
Here we have two bigger steps per epoch, which is basically a parameter here that takes the extreme
shape, which is 28 divided by two.
I'm sorry this is not the extreme shape.
The ActionScript zero zero here is the 60000 sorry, not the twenty eight to sixty thousand.
Divided by the bad size is how much steps per epoch that's this number has 1875 the steps, people,
because we're using a small of that size this time.
So we need to specify this parameter in here when using the tree and its agenda.
So let's run this now, and that's not our results.
So let's take a look.
This may take a bit longer, but we'll see.
So let's wait for it.
OK.
So when that work has finally finished training and you would have noticed some, some things, definitely
you would have noticed.
One is that it took a lot longer.
The train took almost twice the time, and that's because we're doing several things that add competition
alluded to this training process that this was data augmentation, the L to regularize it, regularization
the batch norm and to drop out all of those things contribute to reducing the convergence time.
However, you can see that while 15 epochs wasn't enough to get similar accuracy on the training dataset,
you can see generally the validation accuracy was quite good 89 percent.
Let's take a look at what we got previously, because I don't recall might have been the similar.
Eighty nine percent actually across 90 percent without validation, without regularization.
So while this doesn't look good for regularization methods, however, it does indicate that when using
regularization, you do have to train for more e-books.
I didn't want to make this lesson too long.
But if you have some time to leave it overnight, I recommend you try this for at least 50 books, and
you can see that I'm pretty sure our validation accuracy would be better than if we done without regularization.
So give it a try and you can see an experiment and see how it goes.
One thing you would know is that our validation accuracy was gradually going up.
You may have had a little peek in and drop here, but generally it was going up again, which is a good
sign.
It means that it wasn't really overfitting to training data as much as the previous model, which is
what we want.
So generally, this is a good thing and you can see how the loss was point zero point four zero three
here, although this was actually was quite a bit.
You can definitely see it over fitted at this point here.
So you can you can see generally that the regularization methods have reduced over fitting on our model.
However, we ended up getting the same accuracy in the validation dataset, so it's almost no difference
in the real world, at least after 15 epochs.
But I would definitely recommend you train for more ebooks than this, and I'm pretty sure you will
get a better model out of this than we did before.
OK, so we'll stop there for now.
And next will we're going to do the same thing with PyTorch now?
So we're going to try the PyTorch with the fashion amnesty, the said without regularization and then
train one of which regularization and compare them, as well as we get to show you how we introduce
regularization methods in PyTorch, which is a bit different to us, but they're quite easy.
So I'll see you in the next lesson.
Thank you.