Hi and welcome back.
Now let's take a look at how we add to regularization techniques that we did in Congress in PyTorch
when a similar fashion amnesty to set.
So let's open this notebook and we'll wait for it to load.
There it goes.
So let's begin the lesson.
So just to recap the regularization methods that we will be using as L2 or regularization data augmentation
mutation, which is a data manipulations that you've seen previously dropout.
And that's norm.
So let's begin.
We just do the standard loading of our libraries, which allow us to load the PyTorch models and data
set.
We set our device to GPU.
There we go.
So these things are running and it's going to be finished shortly.
In the meantime, I'm going to talk about how we manipulate our data transforms.
Remember, with Chris, we used our image data generator.
Will PyTorch has that by default.
That's where these transformers are.
And that's what these disclosures are, actually.
And the transformers are basically where we can set all of these augmentations we want to do.
Remember, we were doing.
Remember, we did things like random horizontal flip rotations, grid scaling.
Well, we didn't do those.
And Keros, we used a different set of.
We use some random shifts and skewing insure.
We can do all of those things and pay to watch the all functions to support those things and into transforms
library and pay to.
However, for this lesson, I decided to use some different augmentations.
And then we also visualized these below.
But for now, these are augmentations.
We're going to do random our transformation, which is basically a sharing and warping type effect.
It also has some degree of actually degrees basically said how much you want to skew it by.
It has some translation effect, as well as the sharing effect here.
We're doing college data, which basically changes to hue bit and saturation a bit randomly.
Random horizontal flip, random rotations 15 degrees.
Then we're using grayscale.
We obviously have to do this.
This as well to tend to normalize notice that these are the bottom here.
It doesn't matter in the pipeline where they are, but it isn't.
They do need to be here.
And you can put them at the bottom because it makes more sense to have them there as opposed to having
them in the beginning, just because it's easier to visualize, like what's happening.
And you may have noticed that we have Val here and there are tree in here and Val over here.
Well, this is the transforms as well for the validation data set or test data sets.
It always has to be to see him the same way these days too.
But notice what I wanted you to pay attention to is that we doing to the augmentations individual additional
test dataset.
It's just been done in the training dataset because it's a train time operation.
Next, we create our data ladders, which you've seen before.
Whoops.
I didn't actually run the code above it.
So let's run this block now.
And what this does?
Sorry.
You've seen basically how we actually have to specify the data transforms here instead of pointing it
to the actual transformer having a separate transform.
We use a dictionary to store this information so we can access it like data transforms and access to
key train here, which contains this object here.
And then similarly, we do it for devel, which contains this object here, which contains it transforms.
And we just load the train load said this to for the validation.
Then we create our data loaders here where we shuffle the training dataset, but we're not shuffling
to the test dataset.
We set the number of workers this time to remember.
Previously we used zero.
Now we can know because this club environment has at least two CPU's, we can do this.
And we said about Boutzos here.
So let's move on.
Now, let's take a look at how we add dropout that's known in in the CNN model itself.
It's actually quite easy to do.
So remember, we define our CNN layers here and notice.
I mean, generally, it's a good standard practice to put them in the order you want to do.
But what is actually created here?
But notice we have an end or that's known and and that's known previously in Keros, we didn't have
to set any parameters that specified how many output filters it was or what the input size was.
But in PyTorch, we do, unfortunately.
So we know the output of this filter has to be to a feature map.
So we have to set set to be two here for the batch normalization.
Similarly, for this one here, 64.
And notice we don't have to drop out here yet, but we do introduce a drop down here.
Now the reason for that for us, because the dropout, this dropout layer is going to be reused.
And you'll see it's going to be reused here.
We can call in this multiple times because.
Says just earlier, this isn't actually in sequence, even though it's good practice to put them in
sequence, we can put the drop out below.
It's basically convention in this case would pay too much.
So we did finally is here we have a conflict or conflict with.
That's no, I'm sorry.
That's this one here, and we have the conflict inside of it.
Notice that it's inside of the batch of the batch normal conflict and then we have rlu as activation
function for all of this here.
So that's basically how we set these things up and pay towards this is the input.
This is the output.
And then this is activation.
Then we take this X.
This output gives it a drop out layer and a droplet parameter is specified as point to appear.
And so we then we have another drop out layer at the end of this one where we have come to batch norm,
come through and reload and then drop out layer at the end of all of that.
And then we have a pooling layer which saw Max-Q or flatten function or glue on the next FC layer,
which is a 128 node layer.
And then finally, that returns to the soft max layer.
So let's create this.
So this could be a bit confusing.
However, it's actually quite simple.
And you will get the hang of PyTorch when creating you and see it ends just just by practicing this
thing, just creating you and see and see what happens between it.
It's always good to experiment because this is a lot of code.
This can be overwhelming for many, many people.
So just break it down into pieces.
Step through it.
Experiment.
I python that books are great for experimenting because you can see the results right below.
So don't feel intimidated yet, and they know it and know it will be.
But don't worry about it.
Next, we're going to add L2 regularization.
So how do we do that?
Well, it's actually quite easy.
So how do we introduce L2 regularization in Python?
Well, it's actually a bit different to how we did it in Keros, and that's because we have to do it
within the optimizer itself to sarcastic gradient descent algorithm.
And it isn't.
It's not called L2 and Python, which it's called the weird decay function, and we set it to point
zero zero one here.
That's the L2 isn't L1 regularization by default in this optimizer, unfortunately.
However, we can apply it using this and alternatively if you want, but we wouldn't go into that here.
So let's just create this with delta regularization, which is called way to get right there.
So now we're ready to train our model with all of these augmentation methods.
So let's do it.
Let's set it for 15 epochs.
Is that what we did previously?
Let's see what the setting was 15 bucks, yes, so let's leave this, let's run this and we'll start
our training process and our training process could is exactly the same as it was previously, so we'll
wait for these results.
I'll pause the video for now and come back and then we'll analyze the results shortly.
So I'll see you in a couple of minutes.
Hi and welcome back.
So now you can see a model has finished training.
And similarly, just like the Keros instance, you can see, oh, accuracy isn't as good as the non-regular
rice model.
Now that might be surprising because, you know, I told you that regularization helps get better performance.
However, to achieve that better performance you didn't need, you do need to use more epochs.
So that's basically something you can try on your own.
Let's increase this to maybe 25 to 50 books better yet, and we can assess the model's accuracy.
One thing is good to a generalization is that it's good to experiment a bit initially with outside the
office.
See when your model starts with fitting.
Basically, get a baseline model and try to improve iteratively by introducing one regularization method
at a time.
That's how we keep learning practitioners actually get the best models out.
We don't just throw everything at it and hope for the best.
You need to take experimental strategic approach to improving your models.
Otherwise you can be overlooking something quite simple like you can be.
You probably get the best model by using something simple, sometimes depending on the dataset.
So for now, let's just get overall accuracy, which would be the average of all of these tests.
Accuracy is here.
Eighty nine point nine sheets.
So that's pretty good.
A little bit better than expected.
So let's take a look at our training plots again, and you can see here.
As expected, it looks like it's going to be overfitting slightly higher overall.
This may just be an anomaly.
It could probably continue to go go down.
I'm sorry.
Go up.
Blue is accuracy, so you can.
You can monitor it here.
So I would suggest training this for 50 bucks just to verify that our regularization methods are making
a difference.
And if not, that's just the nature of this.
And basically, that's it for this lesson.
So you would have learned how to add L to regularization by adding it into the optimizer here as this
weird Typekit parameter.
And you would have learned how to easily implement things like drop out and batch Norman where they
need to be placed in the loop.
This is an important part of the code.
Let's take a look at this and remember that when you have a conflict here, this is the input.
This is the first layer.
This is the second.
This is a tool in how that's applied to which it models are constructed.
You don't have to do all in one line.
You can do it line by line where X is equal to self thought into X and then progressively Google to
the batch normally, then reload.
However, this this saves of space and this is sort of like how Keras treats one layer anyway.
Actually, no carrot actually has a separate batch name outside of it.
So just remember those differences anyway.
And lastly, the last augmentation method we added was data augmentation, and that's was done in the
transformation possibly could.
So you can see we introduced all these random data augmentation techniques to manipulate our training
dataset as data is loaded during the training process.
So we'll stop there for now and then the next lesson will continue with the visualization of how CNN's
actually live.
So we'll start with some slides and then I'll dive into the code afterwards for those lessons.
Thank you, and I'll see you in the rest of the course.