So in this lecture, we'll be taking a little detour from Time series analysis, this lecture is all
about extrapolation.
The goal of this lecture is to figure out how different machine learning models perform on data outside
the range of values it was trained on.
This is a question we have to answer for time series analysis.
So picture a time series with a clear trend like a stock price.
Now, clearly, if we split this data into train and test the test, it will take on different values
than the train set.
This is because the TIME series is increasing.
The question to consider in this lecture is can machine learning predict values outside of the range
of the train set?
If the answer is yes, then it would make sense to plug stock prices into machine learning models.
But if the answer is no, then it would not make sense.
In that case, it's probably better to use something like stock returns.
OK, so let's begin.
So we'll begin by importing non-pay matplotlib and a bunch of different models from cyclosarin learn.
The next step is to make our data set.
Basically, it's a sine wave in two dimensions, so the shape of our data is then by two.
The target is just a linear function of the cosine of the inputs.
One thing to notice about our data set is that the inputs are uniformly distributed in the range of
minus three to plus three in both directions.
OK, so from minus three to plus three.
The next step is to plot our data set to see what it looks like.
So basically, it's a repeating sine wave that goes on forever and both the next one, the next two
directions.
The next step is to train a support vector machine.
The next step is to plot the function that our model has learned with the original data points.
Now, this code isn't that important, so don't pay too much attention to the details.
The main thing to focus on is that the range of values for this client is minus three, up to plus three
in both directions.
OK, so we can see that our model seems to do a pretty good job at learning this function, the support
vector machine is pretty powerful.
So this is not unexpected.
The next step is to run the exact same code, except for one small difference, this time we'll let
the range of values go from minus five to plus five.
So we see that when we extend the range of values, our model does not extrapolate well.
Now, sometimes people see this kind of thing and ask, well, how do you know that's not the true function?
The answer is because we've already defined the true function above.
So we know what the true function is and we know that this is not.
So just in case you forgot, we're going to make the same plot again, but this time we're going to
use the true function instead of a model prediction.
OK, so this plot is a plot of the true function, remember, it's a sine wave that repeats in every
direction.
The next step is to train a random forest again, we're going to plot the functional model has learned
within the range of terrain values.
So we see that for the random forest, the predictions are a lot more bumpy and jagged, this makes
sense considering that the random forest is just the average of a bunch of decision trees.
The next step is to check whether or not the random forest can extrapolate again, we've simply expanded
the range of values to minus five and plus five.
So we see that the random forest simply projects outward the last known prediction.
Again, this makes sense, considering what we know about decision trees, there are no more decision
splits beyond the trading values and we know that decision.
Trees learn horizontal lines.
Now, although we'll be studying deep learning in depth elsewhere, we're going to take this opportunity
to try a neural network inside, you learn.
This is called the Multilayer Perceptron or MLP.
Again, we'll start by plotting the learned function only inside the range of train values.
OK, so the neural network seems to fit pretty well, maybe even better than the others.
But now let's see how well the neural network extrapolates.
So, again, we see that the neural network simply extends the prediction out in a straight line.
This will make perfect sense once you know how neural networks actually work.
OK, so in the next portion of this notebook, we're going to see how this concept of extrapolation
applies to stock prices.
Remember what we have just learned?
We've learned that machine learning models cannot extrapolate.
We've just tried a few of the most powerful go to machine learning methods, and we've seen that they
completely fail at this task.
OK, so I'm going to skip most of this code where we download the data and load it in.
So the relevant part starts here where we split the data into training tests, I've just arbitrarily
chosen two thousand as the split point.
OK, so the next step is to create pretend the inputs will pretend that what we'd like to do is build
an auto regressive model with two legs, that is X of T depends on X of T minus one and X of T minus
two.
This is only so that we'll be able to plot this later on.
OK, so it's basically the same loop for both training and test.
The next step is to convert extranet next test into an empire raise so that they are easier to index.
The next step is to make a scatterplot of our data.
So in this plot, XXXIV goes along the horizontal axis index of T plus one goes along, the vertical
axis will make the trains that read and the test set green.
OK, so what can we see?
Well, we can see that the train inputs and the test inputs occupy a totally different area of the input
space.
Not that I've squashed the plot down in the vertical direction just so that we can see it fit on the
screen.
But in actuality, this would look more like a 45 degree line.
Basically, what this is saying is that if you use stock prices as inputs into a machine learning model,
the test data would require extrapolation.
The model only gets to train on the red range of values.
The green range of values is never observed.
Now, sometimes people ask, well, what about scaling the data, for example, by using standardization
or minimal scaling?
Well, remember that scaling the data has no effect on the relationships between the points.
If you divide a time series by one thousand, the values would just be 1000 times smaller, but the
plot of those values would look exactly the same.
And of course, if you're not convinced by this, you're encouraged to try it as an exercise.