In this lecture we are going to talk a little bit more in-depth about training sets versus validation
sets versus test sets.
First let's think about why we would want to split our data up in the first place.
As usual I'm going to return to my dumb as possible picture which is that machine learning is nothing
but a geometry problem.
Imagine that all the data I collected in the past looks like this.
So I find a line that can discriminate between the two classes of data and I can conclude that my classifier
has pretty good accuracy.
Great.
But remember what's the point of building a classifier.
It's to use it on unseen data that I want to classify in the future so what happens when tomorrow's
data looks like this.
Now all of a sudden the classifier is no good.
In fact it's terrible Well that sucks because this is actually the data I really care about classifying.
I don't care about classifying past data.
Why.
Because I already know what the answer is.
So there's no need to classify those data points anymore.
Now you might think this example is a little extreme.
Clearly the data we have seen so far in this course did not behave this way when we get good training
accuracy.
We tend to get good test accuracy but in fact we'll see some data that behaves exactly like this later
in the course
here's another image that should help you understand why we need train and test sets.
The problem is over fitting if you have a complex enough model you will get very good accuracy on almost
any data set but you can see here that although we fit very well to the training data we have a poor
fit on the test data which in this case is the true function that we are trying to model.
This is called over fitting and it's the opposite of what we want which is good generalization there
is a tradeoff that must happen here should you make your model more complex so that you're training
accuracy is improved or should you make your model less complex so that there's a better fit to the
test data.
The problem is if your model is not complex enough then it will perform poorly on both the train and
test data.
So your job is to find the optimal amount of complexity such that it performs well on the test data.
It can't be too complex but it also must be complex enough.
You may also know this as the bias variance tradeoff.
Although this concept is outside the scope of this course you're encouraged to check out the in-depth
series if you're interested in learning more.
So I hope that satisfies your intuition about why we need to have separate data for both training and
testing.
Now you may have heard that sometimes people split their data into three separate data sets the train
set the validation set and the test set.
What's the difference.
Why is it that sometimes we just have train and test and sometimes we have train validation and test.
It's really just semantics when you have a train and test set only your test set is more like a validation
set.
In fact the words test and validation are pretty much synonymous.
They mean the same thing linguistically.
What you are doing when you split your data into train and test is you are actually using your test
step to validate that your model works on unseen data.
So in fact sometimes when people split their data into two parts they'll call them train and validation
sets rather than train and test sets.
In this context the meaning of both is exactly the same.
I hope you're not confused
so in what case well we want to distinguish between train validation and test for this.
I like to think of Kaggle contests although it's well known that Kaggle contests are not a good representation
of how you will do machine learning in the real world.
In fact there have been multiple instances of cheating on the platform.
In any case the way that Kaggle works is this you will have some data that Kaggle gives you to train
your model on.
These are typically labeled as train and test.
Importantly the test set does not come with any labels so you can't compute your models performance
on the test set by yourself.
This is unlike when we split the data into train and tests in our class all you can do with your test
set is make predictions and submit them to Kaggle servers.
Then Kaggle will show you your test score along with the test scores from other participants.
They will list these on what is called a leaderboard.
This is the true test set because it's how Kaggle is testing your model.
But if that's how Kaggle tests your model then how will you test your model.
This is where the validation set comes into play.
Remember it Kaggle.
Generally speaking although this is not always the case we'll give you just that train and test set
you want to perform well on the test set because if you win then you'll make some money.
So you have to make sure that your model performs well on unseen data namely this test set for which
Kaggle withholds the labels in order to do that.
You have to pretend to evaluate your model on unseen data by splitting the train set that Kaggle gives
you and to train and validation sets for this data.
You do have all the labels and so you can calculate both the train and validation accuracy
sometimes.
Although this isn't usually done in deep learning you will split your data randomly into train and validation
sets multiple times.
We call this cross validation.
Here's the basic idea let's suppose you split your training data into five parts.
Then you'll do a loop which runs five times on each iteration of the loop you'll consider four of the
parts to be your train set and the remaining part to be your validation set on each iteration.
You will train a new model on the current train set and evaluate the model on the current validation
set.
When you're done you'll have five different validation scores.
And from this you can calculate the mean variance and so forth to get some statistical measure of how
well your model might perform on unseen data.
The reason this is in use for deep learning is that the data sets are usually so large and training
takes so long that the variance is pretty small and just one validation set should give you a pretty
good idea of your model's performance.
There is a small caveat to the leaderboard.
In fact there are two leaderboards.
The public leaderboard and the private leaderboard the public leaderboard is where your score on the
test set is shown.
The private leaderboard uses yet another dataset and this is the one on which the winner of the contest
is selected.
This is of course to prevent over fitting to the public leaderboard test set the public leaderboard
allows you to submit a solution multiple times so you have some chance of a fitting to it.
On the other hand you only know your private leaderboard placement once at the end of the contest.
So in fact to confuse you even more now there is not just the train set.
The validation set and the test set.
But there is yet another test set
of course Kaggle is not the real world's in the real world.
Your test set is the set of data for which you really want the answer.
But you can't produce using non machine learning techniques.
If you could just write a basic computer program or implement a simple set of rules then you don't need
machine learning.
So when we are using machine learning it's because we really do not know the answer or we can't compute
it practically.
Some basic examples of this are fraud detection spam filtering or object detection we really want these
to work on data that we haven't seen before.
But it's clear that we can't do that work manually.
The next question I want to answer is what is the validation set even for you might be thinking.
Sure I can evaluate the model to get some idea of how I will perform on unseen data but then what.
What do I do with that result.
How is it actionable recall that in machine learning.
We have a lot of choices.
We call these choices hyper parameters.
For example in gradient descent we have to choose the learning rate.
We can't learn the learning rate as you learn more about different variants of gradient descent.
You will have to choose between those variants and then among those variants there are yet more choices
of hyper parameters.
If you have a lot of input features you might want to choose just a subset of the features so that the
model can learn from the important features rather than just the noise.
You also have to choose between different model architectures.
After you learn about basic linear models you'll learn about and ends.
CNN's aren't ends and so forth.
You'll have to choose the number of layers in a network and the number of units in a layer.
You'll also have to choose activation functions.
There so many choices you will feel overwhelmed.
Unfortunately it's not possible to choose these hyper parameters automatically.
You can't just do gradient descent on model architectures.
At this stage your approach amounts to just trial and error.
And so how do you evaluate each of these different choices.
The answer is by using the validation set typically you will pick the model that gives you the highest
validation score and that will be the model that you use on your test data whether that's a Kaggle contest
or pushing your model to production.