So in this lecture, we will be looking at Gargash more in depth and will describe conceptually how
you might extend this model to deepen our own networks.
So as we've gone through this course, you may have noticed one omission, which is that we didn't spend
much time on discussing how these models are actually trained.
This is because this is an application focused course rather than a math and programming type.
Of course, in this lecture we will dive into some of these details, which I believe will help you
understand further.
So with any machine learning or statistical model, there are really two steps of learning how they
work.
The first step is necessary and it's the one we've covered in this course in any other course I teach.
That is, how does the model make predictions?
Or in other words, how do we get from the input to the output?
Often this simply means learning the equation that tells us how to do this.
As usual, this involves your typical mathematical operations like adding, multiplying and so forth.
As you've seen for Gurche, this involves the model parameters, Omega, the Alphas and the bait is
what we have not yet discussed is how are these parameters actually found.
This is the second step now for deep learning.
We've kind of discussed this, but not really.
We know that we call some function called fit and we know that this does something like gradient descent,
which looks like you're traveling down a hill.
That's nice, but that's only about 10 percent of what you really need to know.
If you want to make something like this yourself.
That's not to say understanding those things isn't important, but it does say that the journey to mastery
may be longer than expected.
Of course, this lecture will take you closer to that point.
OK, so the central item of interest when we want to fit or train a model is the objective for deep
learning Arima and many other machine learning models.
This is based on the log likelihood.
Basically, our goal is to maximize the likelihood with respect to the model parameters.
When we find the model parameters that maximize or minimize this objective, we consider the model to
be trained.
Now, it's not my goal in this lecture to explain maximum likelihood from scratch, but you can check
the resources in extra reading dot text if you would like to learn more.
This lecture is more geared towards those who already understand maximum likelihood estimation.
So normally when we want to perform regression, we use the squared error objective.
I know you've all seen this before, so I won't bother to explain this in detail.
Now, if you've taken my in-depth courses before, then you also know that minimizing the sum of squared
errors is the same as maximizing the log likelihood when the distribution of errors is normal.
That is, when you use the squared error objective, you assume that your errors come from a normal
distribution.
What you may not have known is that this also requires you to make another assumption.
This assumption is that the variance of those errors is constant.
In order to understand why this is the case, we're going to review the steps to prove that minimizing
the squared error is equivalent to maximizing the likelihood of the normal.
OK, so in order to maximize the likelihood, we start by writing out the likelihood function, which
makes use of the normal PDF, as you recall, we are assuming that our targets, which form the true
time series, are samples from the normal, where the mean is the model prediction and the variance
is some constant sigma squared.
Also recall that this is equivalent to saying that the target Y of T is equal to the mean Y had of T
plus Epsilon T, where in this case Epsilon T is an ID sample from the normal with mean zero and variance
Sigma Squared also for completion sake.
Well also suppose that Y had of T is some function of the previous plague's and some model parameters
which we will just call theta.
This part isn't necessary, but it helps to frame this in the same way as our auto regressive models
from this course.
OK, so from here we can write our likelihood of function as big of theta.
You should recognize the formula inside the product, which is just the normal PDF.
As you know, the next step is to take the log, which simplifies the optimization.
So as an exercise, you may want to go through these steps slowly by yourself in case you forgot exactly
how this works.
Now, from this log, likelihood, we know that this expression can be simplified.
Firstly, let's recognize that our objective is to find the theta which maximizes this objective.
In other words, the Theta we want to call it theta star is the Amax of the log likelihood.
Now you'll recognize that Theta doesn't actually appear on the right side.
Just remember that Y hat is a function of theta.
I'm just not showing it here anyway.
Once we have this, it's easy to see how this can be simplified.
Notice that the long term does not depend on theta at all.
To Pi is just a number, as is the variance sigma squared.
So if we simply ignored this term, the optimal theta star would still be the same.
The next thing to notice is that the one 1/2 and the Sigma are just the multiplicative constants.
For example, the extreme point of X squared is the same as the extreme point of two X squared.
Multiplying by a constant makes no difference.
So we can remove those as well.
The final thing to recognize is that the minus sign can also be removed.
Basically, we know that maximizing minus X squared is the same as minimizing X squared.
We get the same X either way.
So to simplify this, we can remove the minus sign and change the max to an argument.
Of course, this is just the sum of squared errors as promised.
OK, so at this point, we recognize that in the previous derivation of the squared error, from the
likelihood we made one critical assumption, we assume that the variance of the error is constant.
Of course, this whole section is about exploring what happens when the variance of the error is not
constant.
So in order to understand how Ghazi's trained, we simply have to go back a few steps.
This is our log likelihood again, but with one small difference.
Now I've written Sigma with an argument t this is to signify that Sigma is no longer a constant.
On the left side, we can think of Theta as a collection of model parameters, whether they are for
Y hat or sigma or both.
So this time, because Sigma is not constant, it can no longer be dropped from the Amax.
One small simplification we can make is to drop the two pi since that part is still constant.
We can also drop the one half and the negative signs and turn the objective into something we want to
minimize.
The final step we can take to make this look more like Darch is to recall that our main model is normally
just zero and hence on the numerator we simply have Ypsilanti squared, which is the error time series.
So this is the objective function you would use if you wanted to train a garbage.
So essentially, this is all that you need to know if you want to train your own garbage at this point,
you can plug this objective into any stock optimizer such as LBJ's.
I encourage you to check the CPI documentation if you want to learn more.
Now, as you recall, the objective is not the only thing you need to do the optimization.
You also need to include the constraints to make sure that the variance is always positive.
And of course, many of these stock optimizers allow you to include these constraints.
So this is not a concern.
Also, recall that we derived this objective, assuming that the error is normally distributed.
You can if you want to replace that with any other distribution you like, such as the student T or
the Scooty or anything else.
Now, if you've studied deep learning and neural networks, then your immediate question should be why
do we need to stick with Gurche?
Gurche, as you know, is a linear model and we've seen that linear models can be limited.
So a natural question to ask is why not simply parameterize Sigma T using a neural network?
In fact, this is entirely possible.
So this is an easy but also powerful way to extend Gargash without much extra work that is provided.
You know how to build custom functions using Tenzer floor piter.
One thing that's not immediately clear is how you can constrain the output of your neural network to
be positive.
As you recall, the final layer of a neural network is typically just linear regression, which can
output any real number.
So the way to handle this is to use an activation function that enforces positivity.
For example, the soft plus by using the soft plus, you can ensure that the output of your neural network
is a positive value.
One final topic to discuss in this lecture, which follows directly from what we've just discussed,
is this idea of combining Arima with garbage.
So earlier we wrote our model down as the mean A plus the error.
So it may seem as if we can simply train and arima to learn the mean subtracted from the time series
to get the error and then train Aghajan the error.
In fact, this approach appears in a few Python blogs.
Unfortunately, this approach is suboptimal.
As you saw earlier, the lost function actually incorporates both the mean model and the variance model
at the same time.
What this means is that you find the optimal answer.
Both have to be optimized jointly.
As mentioned, there is no Arima in the Python Arch Library, but there is one in our Yuga, which is
a library for the language.
So if you really need to use a Rhema garage, that would be an option.
One fact to recognise, however, is that it is often the case that an eye one is the most parsimonious
arima for stock prices, as you recall.
This corresponds to a random walk.
When this is the case, there are no parameters to optimize for the Arima power since anyone does not
have any parameters.
So if it is the case that any one is the best model for the price, then there is no difference whether
you choose to fit the model jointly or separately.