In this lecture, we are going to discuss a concept called the walk forward validation.
So to give you some motivation for this lecture, we need to understand why traditional methods won't
work with Time series.
Let's begin with the train test split.
We know that in order to build a model, we should split our data into train and test.
This is how we can avoid overfitting.
Overfitting is when our model appears to perform really well, but only because it's over fitted to
the noise in the training set in the real world.
What really matters is how our model performs on data it has not seen before.
For example, the forecast for next week or whether a new customer is willing to purchase an item.
If a model is too flexible, it might fit very well to the training data, but not that well to the
new data.
Now, there's one downside to the single train to split as we've been doing.
Imagine that in order to build your model, you try out a bunch of different parameters on the data.
You train each model on the train set and test each model on the test set.
Now, unfortunately, what you've really done is kind of like a manual parameter fitting.
Instead of letting an algorithm optimize a set of parameters, you've been doing the work yourself effectively.
Your test set has become in sample data.
You've been using it to optimize some set of parameters.
So one way to mitigate this problem is to evaluate your model using multiple validation sets.
Note that I'm conflating the concept of validation and test sets for the purpose of this lecture will
assume that in the real world, your true test set is the data that lies in the future that your model
really has not seen before.
For example, nonpublic data from a Kagle contest or new customers to your website.
So data for which you truly do not know the answer anything else will consider to be validation data
and will conflate it with test data for simplicity.
OK, so with that out of the way, here's how cross-validation works in a non time series setting,
that is for regular machine learning.
The basic idea is you're going to split up your data into random parts, then you'll repeat this process
at K times.
Each time you're going to pick one of the key parts to be your validation set and the other key minus
one parts to be your train set.
Each time you do this, you'll train a model and evaluate it on the validation set.
Your final score for the model is just the average of the key individual scores.
OK, so hopefully that's pretty simple.
We call this K to cross-validation.
So four time series Kay Foulds cross-validation does not apply.
This is because four time series data, there is a time dependence among the data points.
So if you split the data randomly, you're going to mix future data with past data.
And of course, this is a big no no if you need justification for this.
Think about the fact that it does not reflect how our model will operate in the real world, in the
real world, will collect data and train our model.
The data we collect must be from the past.
Unless you have a time machine.
Since nobody I know has a time machine, it's not possible to train a model with future data.
Therefore, doing so during cross validation is not realistic.
Instead, what is realistic is that we will have past data and we can use all of that past data to predict
the future.
So our new validation method, which is realistic for a time series, is called a walk forward validation.
The way it works is this we're going to pick some minimum amount of data that we will use to train our
model.
This will be our starting point.
Then suppose that we have a forecast horizon H, which can be one or any integer bigger than one.
Then we're going to train our model on the existing training data and validate it on the next stage
data points.
The next step is to then walk forward one step and to spend the next true data point to our training
data and repeat the process that is our train set has now become bigger by one true data point.
And we train a new model on this data and validate it again on the next stage data points.
After this, we continue to repeat this process until we've reached the end of our data set.
OK, so I think that's pretty simple.
Note that unlike cross-validation, this does reflect how our model will operate in the real world.
We will train our model on all available data and then use it to make a forecast for however many steps
we wish to forecast.
And as a side note, it's not necessary to use all available data in the past, perhaps the dependencies
of your time series change over time, such that recent data is more useful than past data.
In that case, you might want to make the training set a constant size window and throw away old data.
This still reflects how your model would be trained in reality.
So it's acceptable.
Another option you might want to consider is that you don't have to walk forward one step at a time.
It's also common to walk forward eight steps at a time so that no validation set overlaps with any other.
But note that if you choose this method, then your window size has to evenly divide the total collection
of points that you plan to use for validation.
So one question that pops up from time to time is, what if I want to use Time's jury split, which
is part of Cycad learn?
So my opinion on this is that it might save you from doing a bit of work under the right circumstances.
At this point, it doesn't help us at all since we are not training a secular model.
So one limitation of these methods is that the model you're across validating must conform to the cyclone
interface, which is not the case for stat's models.
But if you're using SYK, you learn as we will later in this course, then it is a neat option that
might save you from writing a bit of code at the same time, recognize that this method is limited and
this is aside from the fact that it will only work with secular models.
So one way it's limited is that, as mentioned previously, it's possible for us to choose the step
size as we walk forward with Saikat learned you won't have this option and you'll be forced to use non
overlapping blocks.
The Second Way's limited is that you don't get to choose the initial size of your first block, all
blocks will be of equal size.
Neither of these necessarily reflects how you'd work with a time series in the real world.
In the real world, you would probably have the resources to update your model after every timestep.
So a step size of one is probably more realistic.
However, this also depends on how much data you have.
If you have lots of data, then bigger step sizes might be acceptable.
And furthermore, you would probably begin model training with a train set that substantially larger
than your test set so you wouldn't be using evenly sized blocks.
And the last kind of disadvantage of using this method is that it really takes away from your learning
experience.
So when you use this method, you're basically not thinking of all the important details and not thinking
about all the important details.
When you're taking a course specifically to learn about those important details is pretty suboptimal,
in my opinion.
In any case, if you are interested in learning how to use time service split, please let me know on
the Q&A.
Now, just as a final note for this lecture, we will look at how to do walk forward validation in Python,
which is coming next.
However, we will not use this method for every example in this course.
The reason for this is it just makes the code messy and doesn't provide any conceptual benefit once
you already understand it.
So you'll notice that for pretty much all the scripts in this course, we will only do a train to split
at once.
That's not to say you shouldn't use walk for validation in the real world when it really matters.
This goes back to one of the major themes in this course, which is that there are so many options to
choose from.
If we used every option in every script, there would be a combinatorial explosion of things to try.
So as mentioned, one of your major exercises in this course is to mix and match any of the concepts
you learn.
Anything you're curious about, please try it yourself and report your results to the rest of the class.