Hello, everyone, and welcome to this new section in which we're going to look at different strategies
of combating overfitting and under fitting.
These strategies will include data augmentation, as you could see here, drop out regularization,
early stopping, smaller network usage, hyper parameter tuning and normalization.
So far in this course, we've mentioned the terms overfitting and under feeding without really getting
in depth to what this tool actually mean.
Now, for overfitting, we'll consider this plot of the last versus the number of epochs and then precision
versus number of epochs with a loss versus number of epochs.
What will generally have when a model overfitting something like this?
So yeah, we have the training, the validation loss, and that year we have the training loss.
So this is just like some general way of looking at this, though this may take different forms.
Now, that said, as you could see, we have this two sets and your loss versus number of epochs plots.
Clearly we could see that the validation and the training set initially start up with a similar pattern.
Sometimes you may even have the validation which comes up like this.
So you can even have a situation like this where the validation performs even better than the training
set initially.
But generally, when a model over fits, what we'll have will be this kind of model where at some point
the model keeps doing well at the level of the training and starts doing very poorly in the validation
set.
We could replicate this with a precision epoch.
This could be precision accuracy recall or some other metric which we've chosen.
And what we could have would be something like this.
Obviously, you saw in the practice it isn't always like this, but some in this sense.
So you would have this year, you see, you have the training and then you have the validation right
here.
So what goes on is your model keeps on doing very well for the training and then with the validation,
at some point it starts doing even, oh, very, very poorly.
And so the danger with this is if you are considering or working only with a training set, you may
feel like you you more epochs are training over more epochs.
It's a good idea because you keep having this great results like supposing we have fixed this year and
your training is like say 99%, so you have 99% precision score for on your training data and you're
like, wow, this model will do very well in the real world.
But this isn't generally the case because this model actually has been over fitted on your training
data.
Your model has learned instead to modify its weights based on the training data, instead of being able
to extract useful information or some intelligence from the data which has been used in training this
model, the main cause of overfitting is having a small data set and a large and complex model which
contains so many parameters.
Now, if a model like, say, deep neural network has many parameters and you're giving it a small data
set, then obviously it's going to adjust its parameters such that on this small data set, it performs
exceptionally well.
So you may come across a model which has say 99.9% precision or accuracy, just because the data you've
used to train the model has was very small.
Now, in another case, you may have, say, a moderate sized data set and then not just this kind of
model, but a very large model.
So at the end of the day, we notice that there has to always be a balance between this data set and
the model size.
So this means that even if you increase the data size and then you also increase the model size, your
model may still risk overfitting to better understand this concept of overfitting.
Let's take this simple example.
Supposing that you have three subjects to master our school, let's say math, English and sports.
But.
When kids get to school, they are only taught mathematics or they only taught English or only taught
sports.
Let's choose, for example, mathematics.
So these kids get to school, and from the first year to the last year, they taught only mathematics.
It's clear that when you evaluate this kid or when you pick a kid at random and evaluate that kid in
mathematics, the kid would tend to have a better or an above average result in mathematics compared
to kids from other schools.
But when tested on subjects like English and sports, that may not be the case.
And this is because the English and sports weren't taught at school.
And so if you are evaluated on only what you were taught, you would tend to have very high scores like
this.
And then what if we start to evaluate on stuff?
The kids were not taught?
You'll notice that these kids will then start having poorer results because those kids haven't taken
out some time to master English and sports.
And so you may end up with a kid who is able to show, for example, that sine squared x plus Cos squared
x equal one, but can't tell you the past tense of it.
Now it's clear that to readjust this situation, the kids have to be taught all the subjects such that
there is a balance that this kids need.
And obviously this balance will come in a way that the kids now perform better in these two subjects.
And because they have had part of the time they used to study maths allocated for English and sports,
they may perform slightly less than before in maths, but at least what's important to note is this
time around they can now express themselves better in English or practise some sports.
Let's now take this other example.
Supposing we train a model which predicts the presence of a car in an image and then you said you feed
your model with this kind of data.
Then after training your model where you get a test, that model is on this kind of real world data.
It's clear that even if you had very great results with a previous dinner in this, working with this
or testing on this new data set right here, you wouldn't have those amazing results you had previously.
And so it's important that not only should your data be large enough or get as much useful data as you
can, but ensure that your training data represents all looks like your test data or looks like what
you're going to be having in production.
And that's really what matters.
What matters is the fact that you have a model in production which works well or which has a high accuracy
and not a model in your notebook, which has, say, 100% accuracy on your training data, whereas on
production it doesn't perform very well.
Now moving on to under feeding.
It turns out that your model becomes way too simple for it to even be able to extract information from
our data.
So we may have validation data and then training data like this.
And then there is this huge gap between our current loss and the minimum possible loss.
We could also have this at the level of say, let's say accuracy.
So yeah, we could have accuracy and then we have our validation and then training and we have say,
accuracy 100%.
Let's say we have one right here and then our model is still too simple that we just end up say at 0.6
or 60% accuracy in this kind of situation, our data are the relative size of our data as compared to
the model maybe too large.
So you could even have a situation where this data is smaller than what we had here.
So you could be small like this.
But if you model is way too simple, say we have just this very small model, then you may face this
problem of order feeding.
It also turns out that sometimes you may have a situation where you have even a very complex model,
but that model still under fits.
And that's because that model hasn't been built in a way that it could extract useful information from
this data.
Now, if you can recall in the section where we're predicting the car price, we had a situation where
a model we use a simple dense layer that's we use a simple, single dense layer with just two parameters
and we had this fixed data set.
But once we increased our stacked up more dense layers, we found out that we're able to get better.
Training and validation mean average error values.
There are several ways in which we could mitigate this problem of overfitting.
The very first one we'll look at is that of collecting more data.
So it's important to lay hands on as much data as you can.
This data has to be representative of what the model will see in real life, and this data should be
as diverse as possible.
Even after collecting more data to solve this problem of overfitting, we could use data augmentation.
Now, what is data augmentation all about?
Supposing we have this image cell right here, the cell image we have here, which happens to be parasitized.
Now, instead of having just this in our dataset, we could have this image right here modified such
that we now have more data to train on.
So this means that in the case where initially we had, say, 20,000 images, so we have 20,000 images
now after doing data augmentation, after modifying each and every image, we now have a dataset of
80,000.
Now we're considering 80,000 because we're supposing that each image is going to be flipped as we've
just done right here.
So we take this image, we rotate it, we have this other image, the same level obviously still parasitized,
and then we flip again and get this image.
We flip, get this image, flip, get this image we see that we now have instead of just this one,
we have four others.
This actually means we multiplying this by five.
So this is 100,000 since we have it now.
One, two, three, four, five.
Examples.
For this single example we had initially.
It should also be noted that there are many other data augmentation strategies for this kind of image
data.
So apart from flipping as we've just done, we could crop just a portion, we could add some noise to
this data, we could modify the contrast, we could modify the brightness and carry out so many other
operations.
And there is no particular data augmentation strategy which works for all problems.
This means that when you have a particular problem, you will have to try different augmentation strategies
and then be able to select the one which works for your data.
Now, that said, we have drop out to better understand this notion of dropout will consider this simple
neural network right here.
Now.
If you could recall, the reason why we have models which over overfeed is because we are working with
very complex models with many parameters.
Now, in order to reduce the complexity of this neural network, what we could do is take off, for
example, this interaction between this neuron and all those previous neurons right here.
So this means that when training our model, we are only going to consider that we have in this hidden
layer just this two neurons right here.
So all those connections, you have become useless.
Now, this has the effect of simplifying our network as what we have as output.
Now that's after carrying out the drop out operation looks like this.
So we have now connections which look like this.
Now, this particular case is an example of a drop out.
We drop our ratio equal 0.3.
All this is 0.333 as we're dropping out exactly one third of all the connections or rather one set of
all our neurons right here.
And if our equal to third, what we'll be left with will be this.
So we'll take this off and we'll be left just with this neural network to train right here, we see
that we could leave from this very complex model to a simplified model via this drop out operation.
And this has an overall effect of mitigating overfitting.
The next step we could take is that of regularization to better understand regularization.
Suppose we have this model with weights w j so we have c pn weights and this weights are free to take
up any value as we've seen previously.
The fact that this weights can take up just any value may lead to overfitting as now these weights can
be adjusted to fit on the training data in a very perfect manner.
So this means that we could have a model which picks out each and every point like this.
So we have this kind of model and we have this.
So we have this model which picks up every point like that.
Whereas.
If we raise strain.
This waits to stay in a given range, then we may end up with something like this.
So we may end up with a model which looks simplified like this because it doesn't have as much freedom
as this other model.
Now, the problem with this model is if now you put in, you put in this new data, you will find that
this one will try to pull out like this and then your prediction will be somewhere around here.
And so if, for example, we're having horsepower, the x axis, and then you want to predict the price
of a car.
Let's do same year we have the price and then the horsepower.
Then it happens that we have this car with this very high horsepower, this model, because it is over
fitted on this training data will predict this very low price.
Whereas this model, which has generalized on this training data, will tend to predict a more reasonable
price.
Now, it should be noted here that because this model was able to go to each and every point would have
a training loss of almost zero.
Whereas now when we give it this new data, it doesn't perform well.
Whereas with this we wouldn't have a training loss of zero.
But when given new dealer, at least we're going to have reasonable predictions.
So coming back to regularization, our aim is to ensure that this weights because this model can be
represented as this function and this function is made up of this weights.
And so since when we are doing training, our aim is to ensure that we minimize the loss.
Then we could include this weights in the computation of the loss.
This means that we have our loss, which is now equal to the loss we would have normally, plus the
regularization constant times the sum of this weights of each and every weight squared.
Now this is known as LX to regularization, whereas here we have L1 regularization where we're summing
up the absolute value of each and every week.
For now, we're just going to explain how regularization helps in mitigating that problem of overfitting
by restraining those weights in a given range.
So let's have your loss equal.
L does initial loss.
Plus let's call this R.
Now, if our aim is to minimize this loss, then obviously this ll will be minimized and the R will
be minimized.
And so if we're trying to minimize this year or this some in general, then it would have that overall
effect of restraining this weights in a given range, especially as we know that when we square very
large values, these values become even larger.
And so to avoid this.
Our weights will tend to take up smaller values which fall on the smaller range.
This Alta regularization is also known as weight decay.
It should be noted that the main difference between this LX to regularization and L1 regularization
is that in trying to restrain the range of values which these weights can take up, the L1 regularization
has that negative effect of making many of these weights to take up values around zero.
Thus values very, very small or take up many zero values.
So this will lead to sparse models as compared to the LX to regularization.
And that's why in practice we generally use the LX to regularization.
That said, we move on to LX stopping, which we've seen already in early stopping as we have seen.
If we have a model, let's say we have precision validation position and then the training position
which keeps on increasing and then we have this limit of one or 100%.
So we have the precision and we have our epochs.
And then after a while this starts dropping and this is dropping simply because our model is now trying
to over fit on this data it has been trained on.
And so we have to stop training once we notice that the validation performance isn't improving any longer.
So this means that after a certain number of epochs we are going to stop the training.
And what's in this?
Already in the previous section we've seen it both theoretical like this and then we had seen it practically.
Then another thing to do is to reduce the size of the network or use a less complex network.
Our next step will be to properly tune our hyper parameters.
Hyper parameters like by size, drop our rate, regularization rate, and the learning rate can affect
our model and dictate whether this model will fit or not.
Now, if we look at the batch size training with a larger batch, size may speed up our training process,
but working with smaller batch sizes have a regularization effect, which help reduce overfitting.
And so according to your local friends, don't let friends use mini batches larger than 32 for the drop
out rate.
We've seen this already.
Increasing the dropout rate means we are making the model simpler and the regularization rate.
Increasing the regulation rate means we're reducing the effect of overfitting.
And then finally, picking too small of a learning rate may lead to overfitting.
So in general, we have some hyper parameters to tune and they are not only limited to this, as you
may have many other hyper parameters depending on your problem.
Now the fact that normalization introduces extra parameters Mu sigma, which bring in some noise in
the model, has that regularization effect which help reduce overfitting.
So if you include in batch norm in your model, then you could feel free to reduce the dropout rate
since this normalization layer already brings in that regularization effect.
From here we look at ways of mitigating the problem of over under feeding.
So with under feeding, we could use more complex models.
We could also collect more data.
We see that this solution falls on the two that's both overfitting and under.
Feeding your is always a good thing to collect more data or more clean and representative data.
So from here we have you could also improve the training time.
Now note that you could have this model like, let's take this and then we have your you've trained
and then you have the validation.
So here you have the vowel and then you have the train and then you've been training for over a thousand
epochs and then you feel like the model may not perform any better.
Now, several scientists have reported that many times.
They've given up on a model and then come back later after forgetting to stop the training process and
notice that this model kept on performing much better.
So sometimes you don't have to give up on your model.
So you could train or you could increase this training time.
Then again, we have.
Have hyper parameter tuning which could help in making your model more performant.
And we have normalization which stabilizes the training process and leads to better performance in the
model.
We now see practically how the dropout could be implemented with TensorFlow.
So yeah, we have this drop out layer which takes as an argument the rate, the dropout rate, noise
shape and the setting to better understand how and why we need to use seeding is simply in the case
where one reproducible experiment.
So if we want to apply dropout in this layer with a dropout rate of 0.2, then we'll be taken off one
neuron out of this five neurons to make our model simpler and avoid overfitting.
Now, in doing so, we may take this one or this or this one or this or this other one.
So it's a random choice.
Now, if we want to fix this choice so that this experiment can be reproducible, then we can set this
seed so that each time we run the experiment, it's going to be exactly the same neuron which is going
to be taken off.
Getting back to the code, the way we could use this is by imparting the dropout layer.
So here we just have this dropout, we run that and it's fine.
And then we could include this dropout right here.
So we could have here dropout let's let's say dropout rate dropout rates equal 0.2.
Now, you could always increase this rate.
So let's have that.
And then we have the dropout and then drop out or rather rate equal dropout rate.
That's fine.
We could place this out here.
But wouldn't all the one drop out here anyway, you could always include the dropout at this level,
depending on how the model responds to this dropout which has been added here.
So let's add this dropout and then add the dropout right here, bearing in mind that we could always
add more dropout layers and increase the dropout rate or even reduce this rate.
So that's fine.
We run our model.
And as you could see right here, this dropout has no parameters from this.
We look at regular users.
We'll see how to implement the LX to regularize our and the LL one regularize.
So, yeah, we have to cross and then we have this regularization right here.
So if you select LL two, you should follow on this page.
Now, once you get this, you see you have this tab that carries a regularised L two and then you specify
the regularization rate.
So let's copy this out and then get back to our model, the level of our model.
Let's come back to the definition of the curve to DX.
So here we have this kernel regular riser and this kernel regular riser that we use in carrying out
regularization.
So here we have this curve to DX and then we simply specify the kernel regular riser.
We have kernel regular riser which is equal this regularized right here.
Now let's take this off and there we go.
We have this canon regular riser, which is now our L two regular riser.
You could always take this off.
And then from here and you have from tensor flow that carries the regular risers impart L two So that's
it.
You could also import L one so run that.
That's correct.
That that's regular riser, Regular riser OC So that's fine.
We get back to our model right here and then we have just l two.
Now you could have this taken up from here and then you add it out here.
So pattern valid activation ratio and that.
Okay, so we have this and then we add our regular riser and that's fine.
Now you could also do this for the dense layers.
So just right here you could have col regular riser and that should be fine.
So this is how we implement the weight decay with TensorFlow.
Now you could always modify those parameters.
So let's, let's have the k r regularization authorization rate 0.01.
So that's it.
And then we run our model.
That should be fine.
And we could go ahead and retrain this model.
That's it for this section.
Thank you for getting up to this point and see you next.