Hi and welcome to the lesson on understanding, overfitting and generalization, two very important
topics in deep learning because overfitting is a constant problem with deep learning.
So how do we make sure our model is good?
What, what metric or what metrics do we look at?
So we've covered several different metrics to assess model performance, and in reality, there's no
one metric you should be looking at.
It all depends on what you're using your model for.
As I showed you in disease prediction, false positives could be fine.
However, for credit card fraud, a false positive might be annoying for a customer.
Things like spam, you may you don't want to miss important emails to have it classified as spam, so
your spam filters are always set to allow some of the spam to go through just so you don't miss important
emails.
So, as you can see, it's quite subjective to what your model is doing.
So generally, how do we know when we obtain good performance and how do we obtain good performance?
Well, do we use a lot of data?
Do we make deeper, more complicated models?
Do we train for more epochs?
Well, we can do all of those things.
However, there's a quite a few other techniques that improve overfitting, which we'll discuss in the
upcoming chapters.
However, for now, let's define what overfitting and generalization is.
So overfitting is a very common issue that plagues modern development.
It's to be an of all deep learning.
You don't want it over fit to your training data, but what does that mean?
It basically means that your model has been over trained on the training data and ends up performing
poorly on the test of validation data set so unseen data.
This often happens when we don't have enough data or we have used too many features or developed an
overly complex model.
So our model is very, very good at understanding the training data.
However, it basically learned the wrong things to basically predict on new data.
And that's the problem.
It's either a problem of not having enough data or maybe having a overly complex model number for this
use.
But it's a lot of ways we can fix this.
So don't worry.
Let's talk about generalization, though.
Basically, this is a measure of how well our model performs an unseen data models that over to the
training data, are performed poorly on your data.
We just said that.
So that would mean they have poor generalization, and a model that's towards a lot of information has
the potential to be more accurate by leveraging features.
However, this puts it at a risk of storing a lot of irrelevant features.
And I'll give you an example of that.
Let's take a look at this.
Imagine you were given tree images of cores and tree images of fire trucks, and you had tools, basically
a classifier between them.
What if you get this input here?
Red car?
If your model learned to use a color red, as you can see, all these fire engines are red and all these
cars have very little red to identify fire trucks, then you might have classified this car as a fire
truck.
This means this is a very good example of you having over fits your model to the training dataset.
So it didn't learn that it caused different featured model in that color was differentiating factor,
which happens when we don't have enough data or we have an overly complex model that has been learning
the wrong things.
So let's look at a simple example.
We have tree models here, and the models are trained to distinguish between the green and the blue
class.
The model basically would put a plane or a decision boundary that separates these two classes.
So let's look at Model E model is quite squiggly.
You can see it's very good at this here.
Model B is a little more smoother and still is very good, but it misses this class here and model sees
as a straight line here and misses a bunch of these classes here.
So which model would you say is best?
Well, think about it and I'll reveal the answer the next slide.
So this model to middle model is the best performing model, and I'll tell you why.
Firstly, this model is Google Fit because as you can see, Green Dot said move close to here would
immediately be classed as blue when in reality, it probably should be green because you can see it
as like a natural decision line boundary right here.
This blue class can be an anomaly.
We don't want to be predicting four to four anomalies.
We don't want to consider these anomalies as having a big impact on our model because that affects our
generalization.
So we basically want our model to pull back a bit on the screen, Ines, or the basically the sensitivity
to the T of this decision boundary.
And we want this to be nice and smooth.
This gives us good, real world performance.
An undefeated model basically is as a straight line here, and it doesn't take a lot of things into
consideration.
It happens if your model is too shallow or you're using a simple model.
In that case, and this basically describes this year high variance, meaning it's a high and squiggly
ness or variability, and it fits the noises like, as I described and this one has low variance, this
one's a good balance between them.
So let's look at this shot.
What does this tell telling us?
Let's take a look at the legend here.
You have Teslas, which seems a bit unbalanced and going up as the eBucks go, and then you have test
accuracy, which again seems to be hovering below 50 percent, which isn't great.
But look at the trading analysis, the trading lines trading accuracy is close to 100 percent.
In fact, it seems like it reaches a hundred percent here, and trading loss is quite low.
It goes almost all the way down to zero.
This, I mean, this probably wouldn't happen in the real world with most datasets and most models,
but it's basically an embellishment of what can happen in training.
When you're hitting a model here, you can see we have excellent, excellent performance on the training
dataset, but pretty poor performance on the test dataset or losses going up as ebooks go in and our
test accuracy is below 50 percent, which isn't good.
It's basically it was higher when we started.
So you can see that these basically are the good values here and these are the bad values here.
We don't want this to be the opposite.
We want our training, our test loss here to be going down just like training loss.
And we want our test accuracy to be going up just like our training accuracy.
When is it going up?
I mean, going up to eBucks as we train for a number of epochs, more ebooks.
We want all of these metrics to go either up or down accordingly.
So this is an example of overfitting.
So how do we avoid overfitting?
Well, generally overfitting is best done by a combination of methods.
Using a large data sets and reducing complexity is one way.
However, regularization, which is a broad category of techniques where we can truly model complexity
and ensure the model is using the right features to classify the objects.
So let's take a look at this example here.
It's a dog on model classifier.
So we'd like to classify to look at the overall shape of the dog, which means its tail nose is moat
and important features like the overall body type and not to identify dogs and not associated with things
such as which are commonly found in images such as grass or trees and other foot model may predict that
this grass is here has a dog because it sort of see saw grass in all these dog images before.
That's basically a bad model.
And this is a funny example of how to confuse machine learning.
Posted by Dr. Julia Show on Twitter.
This image shows you that this labradoodle this this classifier had a lot of trouble predicting what
was a labradoodle or what was fried chicken with some squeaks, some ice on it.
So this is pretty funny, in my opinion.
Basically, again, this fool is machine learning computer vision algorithms.
These very deep models by just putting some little eyes on Friday looking at anything that it's a dog.
So things to note, things to know about computer vision models.
They're not infallible.
They do have a lot of issues still, and they can be defeated.
But generally, generally, they're quite good at most image recognition tests.
So next, we'll take a look at the regularization methods that help generalization of all models.
There are quite a few methods, so stay tuned for this lesson.
Thank you.