So in this lecture, we will be looking at how to do cross-validation and profit.
We'll start by importing a function called cross-validation.
The next step is to call the cross-validation function.
This will take in a trained model along with values for initial period and horizon.
As you recall, initial represents the number of initial training points, period.
It represents the step size in horizon, represents the number of steps to forecast.
Notice the syntax for these arguments, which are not just numbers but strings.
I'd recommend looking at the official documentation to check for other examples on how to use this.
OK, so the cross-validation function gives us back a data frame, which we've called KVI.
So the next step is to simply print it out.
OK, so as you can see, this is the format I showed you in the earlier lectures we have D why had why
have lower in Y had upper Y itself and cut off recall the format for these dates cut off.
Tells us where we started the forecast from.
D tells us the timestamp for each forecasted value.
Now, luckily, we don't have to manipulate the previous data frame ourselves.
Instead, we can import this function called performance metrics.
The next step is to call this function and see what we get back.
Notice that this gives us back another data frame, which I've simply called PM.
OK, so notice what's in this data frame, hopefully you took notes during the three lecture that you
can refer back to.
Now, the first thing to recognize is that we get a bunch of metrics, including Amisi, our M.S. and
so forth.
Note that there's a message here stating that map has been skipped since the target values were close
to zero.
As you recall, this results in an infinite map.
The next thing to notice is that the horizon starts at six days.
Again, recall that all of these metrics are computed from moving average windows since our horizon
was 60 days and the default window size is 10 percent.
The first value we will see is that six days.
OK, so luckily, profit also has a function to plot the cross-validation data frame called plot cross-validation
metric.
So let's import this function and call it using the metric SMAP.
OK, so we get back a plot showing the rolling window SMAP over the horizon again.
Notice how it starts at six days.
Also notice that the area stays relatively constant over the entire 60 day horizon.
The next step is to call the cross-validation function on our second model, where we ignore the days
which were not open.
The next step is to call performance metrics on DCB to note that we won't bother printing it out this
time.
The next step is to call Plott cross-validation Metrick on Dfki to.
Interestingly, it has a cyclical pattern.
The next step is to do the same thing for Model three, which is where we added us holidays.
The next step is to call Plott cross-validation Metrick on Dfki three.
Again, we see this cyclical pattern.
The next step is to perform cross-validation on model four, which is the model where we added external
regressions.
The next step is to call Plott cross-validation Metrick on Dfki for.
So notice this has a similar pattern as our first model, which included all the zero days, thus it
seems as if what has the most impact is simply removing any days with zero sales or in other words,
when the store is closed.
The final thing we're going to do in this lecture is plot the mean SMAP for each of the performance
metric data frames we got back.
So here we can see that for our first model, we have the worst SMAP, the second worst Esmay is the
one where we added all the external progressors but kept the days where the store was closed.
So it seems that adding all those external progressors only had a tiny effect.
The best estimate comes from the second model where we simply removed any dates, where the store was
closed.
Notice that the third model is slightly worse, which is the one where we added us holidays.
So it appears that adding US holidays did not help.