OK, so in this lecture, we are going to implement simple exponential smoothing in code, as you recall,
this is the same operation as the exponentially weighted moving average.
The main difference is philosophical.
We are now treating this as a forecasting model rather than just a method of calculating a moving average.
In addition, instead of using pandas, we'll be using the stats models library.
So the first thing we have to do in this video is update stats, models.
This is because the current version of stat's models on CoLab is not the same as the one you might install
yourself.
So in order to try and stay up to date with the latest API, we're going to make sure that we run this
command.
Now, of course, when and notice I'm saying when, not if this API changes, you'll have to visit the
documentation to see what has changed.
Remember that with libraries like these, this could happen next week or next month.
So be prepared.
This is all part of being a good data scientist.
OK, so the next step is to import the simple exponential smoothing class.
Next, we're going to instantiate an instance of our new class passing in our passengers data into the
constructor, we'll call this model Se's.
Notice that we get a warning which says no frequency information was provided, so inferred frequency
EMS will be used.
The reason for this is when we load it in our data frame, Panas didn't automatically assign a frequency
to the index.
Also, notice this second warning, which says after zero 13 initialization must be handled their model
creation.
So basically this controls how you determine the first value of the moving average in previous versions
of Stach models.
This was chosen automatically.
In the next block, I'm going to print out the index of our data frame.
And we can see that the frequency is known.
Luckily, we can fix this pretty easily.
We can just assign it DFG Index d'Afrique to the string Ms.
MS means months.
And if you want to know how I figured this out, you can check the link I have left above, which lists
out all the different possible frequency strings that you can assign to date range indices.
Next, I'm going to create another instance of our simple smoothing class with our newly updated data
frame.
I'm also going to set initialization method to legacy heuristic.
This will make it so that it'll work just like it did in the old versions.
As you can see, we no longer get the above warnings.
The next step is to call the fit function on our model, as I mentioned in the theory lecture, we are
going to use a fixed alpha and we're going to set optimized equal to falls.
This is just so that it does the same calculation we did previously, which allows us to compare the
outputs.
So let's assign the return value to Rez.
Next, I'm going to print out Rez so we know what we are working with.
As you can see, it's a whole winter's results rapper, so if you want to look up the documentation
for this class now, you can.
The next step is to call the predict function on a results object, since I want the predictions for
our entire dataset.
I'm going to pass in a start date equal to the first row index of our data frame and end date, equal
to the last row index of our data frame.
As you can see, this returns upand a series.
In the next block, I'm going to call the same function again, but I'm going to assign it to a new
column in our data frame called Se's.
Next, I want to demonstrate that the predicts function, since we called it for only in sample dates,
actually returns the same thing as the fitted values attribute.
So in the next block, I use the function and I'll close passing in these two strings.
Since the result is true, that means the predict function returns to the same values as the fitted
values attribute.
Next, I'm going to plot our data frame.
Note that this is the entire data frame, including everything we put in there previously.
Now, right away, you should notice something strange.
The results from our CBS model are different from pandas.
They seem to be shifted up by one.
Why is this?
Let's do a dive head to get an idea of what's happening.
As you can see for both women and says the first value is 112, which is just the first value of our
Time series for either WMA, the next value is one thirteen point two.
But for Se's, the value 112 gets repeated again.
After that, we have won thirteen point two.
So what's going on here?
Let's try to shift the sex column back by one to check if the rest of the values line up.
In the next block, we're going to plot WEMA against X1.
As you can see, the values do indeed line up as suspected.
However, you'll notice that I have this comment in here in which I tell you very loudly, no, do not
do this.
In fact, this is wrong.
Now, I won't name any names, but I've seen a few other people who do this who happen to be very popular
among beginners.
So why is this wrong?
Recall that the forecasting model is defined as slightly differently from the traditional IWM.
As you recall from the theory lecture, the forecasting time index is actually moved up by one step.
That humor is represented by the level.
But the prediction why hat is actually assigns the level at the previous time step.
In other words, the SES model should be lagging behind by one time step.
So what we had plotted originally was correct.
You'll see in our later examples why this is true.
Remember that if we shift this, then we have to shift everything.
But you'll see that with whole winters', the full model, there is no need for shifting because it's
actually the correct model and it will match up very nicely.
If we shift this, then we have to shift that to be consistent and you'll see that would be completely
wrong.
All right, so the next thing we're going to do in this lecture is to treat this more like a machine
learning problem, that is we're going to split up our data, set into train and test, and we're actually
going to do a forecast.
So I've sat and tested 12 and I've set the train set to be everything up to just before the last 12
data points.
I I've therefore set the test set to be the last 12 data points.
Next, I'm going to recreate the simple smoothing object, but this time with only the train set.
Again, I'm going to give this the variable name Se's next I call the fit function.
Notice how this time I'm not going to pass in Elfa or set optimises equal to false.
What this function will now do is find the best alpha so that it minimizes the squared error over the
train set.
The next step is to create indices which we can use to index our data frame.
We need this if we want to index only the train points or only the test points one at a time.
So to do this, we're basically going to create a one, Zeray, of booleans.
It's going to be true for the rows, which corresponds to the correct data set, whether that's train
or test.
So for the trains that we do, DFG index less than or equal to that train of minus one.
That is to say, any row where the index is less than or equal to the last training point belongs to
the train.
So of course, this makes sense since the data set has been split in chronological order.
And using that, we know that the test index is just the opposite.
So we use greater then instead of less than or equal to.
Next, I'm going to assign the predictions from our model back to the original data frame ADF to do
this.
I'm going to add a resident fitted values to the data frame, indexing the rose by train ATX.
This will be assigned to the column Se's fitted.
Next, I'm going to call resident forecasts for NS has time steps.
I'm going to assign this to the data frame under the same column in the final and test rows.
Next, I'm going to plot the passengers and SES fitted columns.
As you can see, the forecast is indeed a horizontal straight line, as promised.
Note that again for the train set.
The prediction looks like it's lagging behind by one step.
You, again, may be tempted to shift this back so that they line up perfectly.
But remember, this is not the correct thing to do.
A simple reason to remember why that is, is because you know that the model stores the dates, so we
have to presume that the model lines up those dates correctly.
Now, if you did shift it back philosophically, think about what that means.
That means this simple exponential smoothing model makes it nearly perfect predictions, which means
that there is no further work to be done.
We don't have to model the trend.
We don't have to model the seasonal component.
And of course, that would not make any sense.
Furthermore, looking at this plot, we can try to guess what the value of Alpha might be when our Alpha
was zero point to the one we set manually, the IWM looked very smooth and didn't track the original
signal that fast.
On the other hand, for this fitted alpha, we can see that the model is reacting very quickly to the
original Time series, which would indicate that it cares more about the most recent sample and less
about the previous average.
Therefore, we might guess that this alpha is very close to one rather than being closer to zero.
In the next block of code, we check what the value of Alpha is by printing resident Prem's.
As you can see, the result is, in fact, very nearly one, that means our fit and model is simply
doing nearly the night forecast, or in other words, it simply copies the last known value in the series.