Okay, So in this video, we're going to use our hands to do time series forecasting.
Okay.
So what's on the agenda for this lecture?
So this lecture is going to do the basically the same steps as we did with the previous deep learning
architectures, where we do the one step forecast, the incremental multi step forecast and the multi
output multi step forecast.
Okay.
And so just as a reminder, there are many configurations we can try.
And so these are some examples of what you can do.
So the most basic method is to take the final hidden state from the Arnon and then pass that through
the final dense layer.
The second method is to take all the hidden states use global max pooling, and then this will give
us a single hidden vector which we can then pass into the final dense layer.
And then the third option is to stack multiple almonds together.
So in this lecture, we're going to use LCMS.
But be aware that you can exchange LSB SMS with or use at any point in the script.
Basically with no changes in code.
So you might want to give that a try.
But otherwise all the imports are the same.
Okay.
So the next step is to update psych and learn so that we get the latest metrics.
Okay.
And so from side, you'll learn we're going to import the metric.
All right.
So the next step is to download the airline passengers CSV.
And you've seen all this before.
So we read in the CSV using CSV, and then we take the log of the passengers column and then we split
the data into train and test.
So the last 12 data points will be used for the test set.
Okay.
And so for indexing purposes later on in this notebook, we're going to want the train index and the
test index.
So we want a Boolean to tell us which indices belong to which data sets.
And so the next step is to calculate the first difference of the log passengers.
Remember that you don't have to do this.
You can always try this script without taking the difference first and then compare the results.
Okay, so the next step is to make our supervised data set.
So we begin by dropping any names from our difference to column.
And then we set our past time steps to ten, and then we create our empty arrays for X and Y.
And then we loop through the time series, and then we assign past data points to X.
And the next single data point to Y.
So this is our data set for making one step forecasts.
And so note that at the end of this we reshape X.
So originally X is just NW by DT because there's no feature dimension, but our ends expect an input
of shape N by DT by D.
Even though D is just one.
And so you can see we have 133 data points for both X and Y.
Okay.
And so the next step is to split the data into train and test.
Okay, so the next step is to build our RNN.
So it's very simple.
So notice this looks pretty much exactly like a regular arm, except that we've replaced the second
layer or the second object with LCM instead of dense.
And again, feel free to try different ana layers like the symbol N or the G or U.
And so important to notice that the input shape is t by one.
So t because we have t time steps and the feature dimension is just one because we only have one time
series.
And so for the lshtm now a lot of people will not a lot, but some get confused about what this number
means.
So remember that this number 24 represents the size of the hidden vector.
And so one common mistake I see a lot is confusing.
T with this value, which we've called M in the theory lectures.
So that's M So M is not the same as t.
M is the size of the hidden vector h, whereas t is the length of the sequence.
So that's how many vectors we have.
Okay.
And some other comments with this.
So the lshtm by default.
So notice how we haven't specified any activation function.
And this is because the LSHTM uses the ten fps by default.
So it's kind of a weird layer where all the other layers just do nothing.
The Lshtm uses ten by default and also by default.
This returns just the final hidden state.
So we're getting at Big T, and so all that's implicit.
We don't have to specify that directly.
Okay, So I just wanted to make an extra quick note that if you want to make use of the GPU while you're
using an LZ team, you need to follow certain requirements so you can check the TensorFlow documentation
for this.
But basically it's that you have to use the ten inch activation, which we've mentioned is the default.
You have to use the sigmoid as the recurrent activation.
So you saw those in the equations from the theory lecture and you must not use dropout.
You must use unroll equal the false and you must include the bias term.
And the rest of this stuff is not really relevant at this time.
So just keep that in mind.
So if you were thinking about changing the activation to something like the REL, you keep in mind that
this would prevent you from making use of the GPU.
Okay.
And so the final layer, we have just a dense one because we are making one prediction.
Okay, so let's do a model summary to see what we got back.
Okay, So it's pretty straightforward.
So our input is of shape MN by DT by D as expected.
And by the end of the lshtm we have N by 2424, because that's the size of our hidden vector and we
only have one hidden vector at the end of the lshtm.
So it's basically NW by DX.
And then once we have that, we can pass it through a final dense layer and then we get NW by one.
And that one is the prediction for the next time step.
Okay.
So the next step is to call compile with the losses mean squared error and optimizer is Adam as usual.
And the next step is to fit our model on the train set and then validate on the test set.
We'll do this for 100 epochs.
Okay.
And so while this is running, I'll just talk about briefly why did we choose 24 hidden units?
And the answer again is this is just an arbitrary number, right?
You have to test different numbers and see what works best on the out of sample data, of course.
So there's no formula for this, as some people might expect.
Okay, so this is our training loss and validation loss.
It's always a good idea to plot the loss to see that it converged.
So it looks like both the train loss and the test loss have converged.
Okay, so the next step is to set the first T plus one values to false in the train idea.
And this is because they are not predictable.
So we're not going to make any predictions on the first few values because the model doesn't have enough
pass values to make those predictions.
Okay.
And so the next step is to call predict on both the train and test set and then to flatten those outputs.
Okay.
So this gives us PE train and PE test.
And so the next step is to compute the difference to predictions.
As you recall, this requires us to have the data at the previous time step.
So we shift the log passenger is backwards, and then we save it to a variable called preve.
Okay.
And we also keep track of the last known train value.
So this is the value we're going to accumulate from when we're making our multistep predictions.
Okay, so the next step is to make our one step forecast.
And so remember that because we have a difference, the data, we need to add it back, basically.
So the prediction is not just the prediction because these are the difference predictions, but we need
to add it back to the previous value.
So it's basically the previous value plus the delta that gives us our current prediction.
And we do this for both train and test.
And remember, these are just a one step forecast.
So that's why we can use the previous value without any philosophical trouble.
Okay, so the next step is to plot the one step forecast.
And so as you can see, it looks pretty good as expected.
So the orange line is the train prediction and the green line is the test prediction.
Okay.
So the next step is to do our multi step forecast.
And so remember that we have to do this incrementally.
So we create an empty array to store the predictions.
And we begin by grabbing the first test input.
Okay.
And so once we have that, we do a loop and we loop until the length of our prediction array is less
than end test.
So once it's equal to end test, then we've made enough predictions.
And so inside this loop we call models predict.
And so what do we call models predict on?
So we call models predict on last x, which is going to be updated within this loop, by the way, and
we reshape it to be of size n by t by DX.
And I'm just being lazy here with the minus one because minus one is a wildcard that fills in whatever
value needs to go there.
Some people believe that you should be more explicit, but I think this is fine for now.
So you could put T in here if you wanted.
It would be just as easy.
But just remember this RN, ht and DX and both RN and DX or one because we only have one sample and
we only have one individual univariate time series.
Okay.
And so once we have our prediction, we grab the zeroth element and we'll call that P.
So remember the model dot predict always returns an array.
And so once we have our prediction P, we append that to our list of predictions.
And then once we have done that, we make our new input.
Okay, so last X.
So we're basically shifting everything backwards and adding the newest prediction to make the prediction
for the next time step.
Now, you've seen this before, so I won't spend much time explaining it again.
Okay.
And so the next step is to save the multi zip forecast to our data frame.
And so just to recall, how do we make the multi step forecast, the incremental multistep forecast.
So we take the last train point.
Okay.
So this is the last data point that we know.
And then we do the cumulative sum of the different predictions.
And so you saw the formula for that in the theory lecture.
So you understand why this works.
Okay.
And so in this plot, we're going to do the multi step forecast and the one step forecast together.
And so notice we're not bothering to do the train predictions anymore.
Okay.
So you can see that they're both pretty close.
But of course, the one step forecast is slightly better because it uses the true past data.
Okay, So the next step is to do the multi output forecast.
So we're going to start by making the data set for this.
Okay.
And so just a reminder about how this works.
We're going to have multiple time steps for our input and we have multiple time steps for our output.
So our model is going to predict 12 different values simultaneously.
And so you've seen all this code before, but just really quick, we create our X and Y again and notice
t, x and t y may be different.
And then we loop through the time series up to a point so that we don't go out of bounds.
And so inside this loop we grab two data points and we make that into X, and then we grab T data points
the next T data points, so offset by X and we append that to Y.
And so outside this loop we have to reshape X and Y, and so X as usual is going to be n by T, by dx,
whereas y is now going to be n by t y.
Okay, So this is like a neural network that outputs multiple values for say, classification.
It's just that instead of n by K, we have n by t.
Y.
Okay.
And so the next step is to split our data into train and test.
And the next step is to create our new r.
N.
So notice that for this r.
N.
We're using return sequences equal to true.
And so this gives us all the hidden states.
So h1h2 all the way up to T And then we use global max pooling to choose the best values of those of
each component of the hidden states.
And so that gives us a single vector.
And then once we have that single feature vector, we pass it through a final dense layer with t outputs.
And so there's no reason that we had to use return sequences equal to true and global Max pulling on
this example, whereas we didn't use it on the previous example.
Right?
You can mix and match any way you like, but obviously we don't want to try every single combination
since that would be very tedious.
And okay.
And so the next step, I believe the training was a bit erratic.
So I decided to create a checkpoint to save the best model.
So we have a model checkpoint, and this will save the best model according to the validation laws.
And so the next step is to compile using the same loss, the MSI and optimizer atom as usual.
And then we're going to call model fit for 300 epochs.
And notice we're passing in our new data X train and we train for multi output.
And we also have our callback to save the best model.
And.
Okay, so let's wait for this.
Okay.
So, as usual, we're going to plot our loss per epoch.
Now case of the last epoch looks good.
Maybe you could have trained it for longer, but it does look like the test losses may be creeping upwards.
Okay.
So the next step is to load our best model.
So remember that we save the best model using a callback.
Okay.
And so the next step is to call models predict to get the predictions for the train set and the test
set.
Okay, so the next step is to check the shape of Petrie and test just as a sanity check and to make
sure we understand what's going on.
So the train predictions, we have 121 train predictions, and each of those will make a prediction
for all 12 time steps in the future.
So obviously there are some redundant predictions, but they're using different past data.
Okay.
And so for the test set, we only have one prediction.
But for 12 different time steps.
So basically, instead of 12 separate predictions, we have one prediction with 12 prediction heads.
And so this is just to make sure that our data is in the right shape.
Okay.
And so the next step is to save the multi output forecast or data frame.
And notice again, we use the cumulative sum because we've predicted the differences and not the actual
time series values.
So that's the last known data point plus the cumulative sum of the difference predictions of the deltas.
Okay.
And so the next step is to plot all the forecasts at the same time.
So basically we're just adding the multi output forecast to the forecast we already made.
Okay.
And so it's hard to see, but it looks like the multi album forecast, which is in red, is closer than
the other ones.
But we'll use actual quantitative metrics later on in this lecture.
And so just as one final experiment to show you the different possibilities of what you can try is we're
going to add another lshtm layer.
So we're going to stack Lshtm layers together.
Now, people always assume that this is just going to be better.
More is better.
But maybe you'll be surprised at the result.
So we have for this example, I've chosen 16 hidden units arbitrarily.
And so why did I do that?
No particular reason.
Well, maybe one reason for this is because when we add less layers, we used more hidden units.
And now that we have more layers, we can use less hidden units.
So just to keep from blowing up the number of parameters, Okay.
But otherwise, you know, you can use whatever number you like here, whatever you feel is appropriate
and whatever you want to test.
And then we're going to use our global max pooling once again.
So just to be clear about what's happening.
So the input is a bunch of vectors.
So two vectors that make up a time series.
And then we pass it through.
The first LCM with return sequence is equal to true.
So we again get a sequence of more hidden vectors.
So this is again, vectors of length.
So we still have x head in vectors.
And so when we pass this through the second lshtm, again, we're passing in rt x hidden vectors and
we get back to x head in vectors again.
And it's only after we do global max pooling is that we take those x head in vectors and then we shrink
them down into one hidden vector.
Again, you can see you can do that as many times as you like, right?
Because the input is kind of the same as the output input is x head in vectors.
Output is head and vectors.
Okay.
And so, again, we're going to create a checkpoint to save the best model.
And we're going to compile using the same arguments and we're going to fit with the same number of epochs.
Okay, So it still goes pretty fast.
Okay, so that's done.
And we can now plot the loss again.
Just to make sure that our model has trained successfully.
And it does look that our loss has gone down.
And again, perhaps you could have done more epochs if you wanted.
You can try that on your own if you like.
Okay.
And so, again, we're going to load up the best model that was saved before during training.
And so the same step is above.
We're going to call models predict and we're going to grab the relevant indices.
And again, we're going to do the cumulative sum to get the prediction into the right format.
So we'll save this as the column multi output to.
Okay.
And so this time we're going to plot a thing, just the multi step and the multi output forecast for
the second one.
So not all the forecasts, just the incremental multi step and the second multi output forecast.
Yeah.
So I think if you did too many with the plow would be too crowded.
So we're just doing these two and you can see that the second multi output forecast is a lot closer
than the incremental forecast.
Okay.
And so the final step is to compute our metrics for each of our experiments.
And so notice we're not bothering with the one step because that's not really comparable, because it
uses true past data versus all of these methods do not use true pass data.
Okay, so let's run this.
And we see that we get the best result with the single LSB is 0.0054 versus when we had two LSB LCMS,
it was 0.00579 OC.
So this shows that just adding more layers isn't necessarily better, as some people assume.
OC And it also could be because we didn't choose the right number of hidden units.
So I will let you try that on your own as well.
Okay.
And so as a final thing for this lecture, here are some exercises, things for you to think about.
And as always, you can try this in the code as well to confirm or deny your predictions.
So think about.
Is our improvement.
So notice how the model improved after we added global max pooling, but we also changed other things
at the same time.
So when did we add global max pooling?
So we initially added global max pooling.
So our first experiment was without pulling, we just took the final head state.
And then from that we did both the one step and multistep forecast.
And then we made a multi output data set.
And so we use global max pooling for the multi output forecast.
So really we changed multiple things at the same time.
So it's not clear why that model was better.
It's better because it's predicting multiple outputs at the same time.
Or is it better because it's using global max pooling?
So think about that and also think about does multiple LSB layers help or hurt?
And so remember that this also depends on the number of hidden units you choose.
So it's not just one thing.
Okay.
And so another question to consider is, do you think different thing is necessary?
So I know a lot of people who come to this course, they came here because they wanted to learn how
to use LCMS for Time series forecasting.
They didn't really care so much about Arima or ETS or and NS or CNN's, but we've seen that those models
work pretty well.
Okay.
But we also noticed that they all don't work very well when you don't use different thing.
So what I'm getting at is people often believe that because lshtm and rans have been so powerful for
NLP that they must be really, really powerful for Time series as well.
And you don't have to do anything like different thing because you know, Rand's are just magic, right?
They'll automatically learn the pattern.
This is what some people believe.
So if you think this is true, if you believe this is true, I would recommend trying to not do different
thing and seeing if that gives you a good result.
So just give that a try if you think different thing isn't necessary.
And there's another thing to consider.
Does logging work or is it also unnecessary?
So try not taking the log transform of the time series and see if it's okay.
Ken.
So another question that you might want to consider is do you think including more past legs would have
been useful?
So this data set obviously is very small, so you can have like 100 past data points.
But if you have a larger data set, you might want to give that a try as well.
OC compare, say ten passed data points versus 100 past data points.
So these are always experiments you have to do, and the answer is going to be different based on which
data set you are using.
And here's another thing to consider.
When you're doing this in the so called real world, you might want to consider using mock forward validation
to optimize those parameters.
For example, the number of hidden units, the number of hidden layers, whether you should use global
max pooling or not.
So all of these questions, they can be answered by doing walk forward validation.