In this lecture, we are going to circle back to the earlier part of the previous lecture, which is
how do we define the best model, we know that Otto Arima will find the best model out of some set of
models.
But of course, in order to make any sense of that, we need to know what it means for a model to be
the best, because these models come from the statistics literature.
If you come from a machine learning background, you will recognize that machine learning people might
approach this topic differently.
However, Arima is a statistical method.
And so generally, when we look at how these libraries work and what they do, they will be using techniques
founded on traditional statistics rather than machine learning.
What is somewhat interesting about Otto Arima is that we end up coming full circle.
As you may recall, the reason that we use methods such as the Akef and the pickoff is because we want
a more direct way of choosing hyper parameters than simple trial and error in comparison.
Trial and error seems like kind of a very naive approach.
And yet with Otto Arima, that is exactly what we do.
It turns out that the method of manually looking at the HCF in the Sieff does not always lead to the
optimal answer.
Otto Arima does a more exhaustive search and therefore has a better opportunity of finding the best
answer.
Now, one method you may have thought of is grid search that is searching through every possible set
of hyper parameters in a grid.
For example, if you had it to hyper parameters to search for, this would be a two dimensional grid.
For example, you might want to try different combinations of P and Q from one to 10.
So it's one one, one, two, one three and so on.
If you had three type of parameters to search for, this would be a three dimensional grid.
A grid search would be simply to look through every possible position on the grid to find the best combination
of high parameters.
Now, although Auto Arima gives you the option to do this, this is not the default behavior.
Instead, Auto Arima uses a stepwise algorithm to more intelligently find the best set of hyper parameters.
It's true that computers are so fast nowadays that they allow a function like auto arima to be possible.
However, a full grid search can still be quite slow.
Therefore, we typically use the stepwise algorithm as the default.
All right, so when we run out Arima, it's going to use some criteria to evaluate each model.
The model that gives us the best value will be the model chosen to sort of equally good evaluation criteria
or the AIC and the Bisi AIC stands for a information criterion and Bisi stands for Bayesian information
criteria.
The intuition behind both of these is the same.
When we are building machine learning models, we often have to make a trade off.
This trade off happens between a model complexity and model accuracy.
For a model, complexity means increasing the values of P and Q.
Recall that P is the number of past data points to include in the model, and she was the number of
past errors to include in the model.
You can imagine that as we add more and more terms, the model will get more and more accurate.
In fact, if you studied linear regression with me in the past, then you know that even adding completely
random noise will increase the accuracy of your model.
This is not good.
How do we know when enough is enough?
How do we know when we've overdone it?
In statistics, the answer is to penalize the moral complexity again, if you've studied with me in
the past, then you know what I'm about to discuss.
If you haven't.
That's OK.
But feel free to ask me about this on the Q&A if you want to learn more.
It turns out that the laws function when we optimize these Arima models is the negative log likelihood.
It also turns out that for the most part, minimizing the negative likelihood is equivalent to minimizing
the squared error.
So this doesn't contradict anything I said earlier about minimizing the squared error of the predictions,
the log likelihood is more general, however, since it can account for variance.
Now, if we only look at the log likelihood, we might end up overfitting.
So what we do is we add a penalty term to the negative likelihood.
The main difference between the AIC and the Bisi is that this penalty term is computed differently.
So just in case you're curious, the AIC and the BSI are defined as follows for both of these, you
will have two times the negative likelihood.
So in these equations, l represents the likelihood.
And so each of these contains the term minus to log out for the AIC.
The log likelihood is penalised by adding two times the number of parameters in the model for the Bisi.
It's penalized by adding the number of parameters in the model times, the log of the number of data
points.
So they both do the same thing just slightly differently.
Auto Arima happens to use the EIC by default, although it's often said that both of these usually lead
to the same answer anyway.
For some reason, statisticians always discuss both of these simultaneously, even though you always
end up having to choose just one.
All right, so now that we've discussed the statistics way of doing model selection, I want to briefly
discuss how a machine learning person might go about this task.
Note that this is entirely theoretical at this point.
We are not going to use this method, and I only mention this out of interest.
In machine learning, we often don't care that much about the number of parameters in a model.
One reason why this is, is that you might be comparing different kinds of models.
If you're comparing a decision tree to a neural network, for example, they are not really comparable
in that way.
Another reason is for modern methods such as deep learning.
It often doesn't matter in the pre deep learning era.
That is, before people even came up with the phrase deep learning.
You will see a lot of papers on neural networks talking about this idea of comparing the number of parameters
to the number of samples.
This makes a lot of sense when you look at linear regression because the actual data Matrix X has a
number of rows equal to the number of samples and number of columns equal to the number of parameters.
So this makes sense when you consider linear algebra and the solution for linear regression and so on.
People used to extend this thinking to neural networks, but today we have found that neural networks
don't actually behave badly when you have many more parameters than samples today, you can have neural
networks with billions of parameters that cost the equivalent of millions of dollars to train.
So it's not really model complexity that we care about in machine learning.
In machine learning, what we really care about is the ability to generalise, that is to say we don't
want our model to be accurate only on the data that it was trained on.
We want it to be accurate for data that it hasn't seen yet.
This is important for pretty much everyone that does machine learning.
For example, if you're building a recommender system, those recommendations will be going to people
who have not yet seen the movies or purchased the products that you were going to recommend.
If you're building a fraud detection system, you want to detect future fraud, not pass fraud, which
you already know is fraud.
If you're building a time series forecasting model, you want the forecast to be accurate, not just
the data from before the forecast.
For many models, this is highly correlated to model complexity, which explains why model complexity
was something people care about.
In the past.
You would see plots such as the following weather test performance starts to degrade when the model
becomes too complex.
So given this information, what might we want to do instead, if our task is to choose the best model?
Well, why not simply check the out of sample accuracy or the test accuracy directly?
In other words, why bother checking the AIC or the Besi when we can evaluate each model according to
it's out of simple accuracy.
This seems like it will more closely do what we want to do rather than adding some penalty term to the
sample accuracy, which we already know is less relevant in the scenario where we are using out of sample
data to choose hyper parameters.
We would call this the validation set rather than the test set.
You might use methods such as cross-validation to choose your hyper parameters.
So that makes sense when what you care about is good accuracy.
However, one reason to use the AIC and the Bisi is because we really are trying to achieve the simplest
model possible.
As you recall, one of our main motivations is modeling the data and explaining how it arose rather
than making predictions.
If this is our motivation, then it makes sense to want the simplest model possible that can adequately
explain the data that we saw in statistics.
Sometimes we call this parsimony.
We say that we want to find at the most parsimonious model.