In this lecture we are going to look at a slightly more advanced topic in PI talk which is creating
custom lost functions.
In addition we're going to apply this new knowledge to build a new kind of neural network one that not
only estimates a function but also estimates the uncertainty in the function prediction previously.
We spent the entire course only looking at neural networks that act as function approximations for each
input.
We get some output prediction but you may have noticed something curious.
There is in fact a discrepancy between a regression and classification with regression.
We only get one answer.
The model's output is our best guess for the target given that input.
But with classification we get a probability distribution.
This probability distribution gives us some sense of the confidence in that prediction.
For example if we're doing binary classification and the output is zero point nine nine then we might
conclude that our model is very confident in that prediction.
If the model outputs zero point five five then it would still make the same prediction.
But intuitively we know that we are less confident in that prediction.
You can apply the same reasoning to soft Max outputs.
So given this knowledge you might wonder why don't we output probability distributions when we're doing
a regression
so if you've studied regression and the mean squared error more in-depth then you know that doing regression
with the squared error is nothing more than maximum likelihood estimation with maximum likelihood estimation.
We start with the likelihood.
Take the log of the likelihood and then negate the likelihood to make it a loss to minimize rather than
a utility to maximize in this form.
We say that T which is the target is the random variable which we assume comes from a Gaza and distribution.
Why is the model prediction which is the mean of the distribution and X is the input to the model we
call theta.
The collection of model parameters.
So this can represent a neuron that work linear regression or any other regression model.
Doesn't matter
as you've seen the usual first step is to take the log of this function at this point your gut instinct
should be this.
We know that we want to end up with the T minus Y squared term and all the other terms should drop off.
That's what gives us the sum of squared error or the mean square error and we know that we can do this
because the log of 2 pi sigma squared is constant constants don't affect the gradient of the loss.
So whatever model parameters maximize this log likelihood would also maximize the log likelihood minus
or plus any constants
but here's the major difference in this lecture in this lecture.
We are not going to make Sigma a constant.
Instead we can ask what if sigma is also a function of x.
So now this term can't be dropped from the objective function because it is not constant.
What this means is that in this scenario this whole thing will be our last function instead of just
the squared error.
And as usual we can take the mean instead of the sum which is equivalent to having a Monte Carlo estimate
of the expected value.
The next question you should have is OK now we're trying to predict two things the target and its corresponding
variance.
How can we do something like that.
Well why not just have a neuron that work with two output nodes.
Let's remember that it doesn't matter what the meaning of these two output nodes is.
We've seen that a neuron that we can end with a linear layer with any number of output nodes.
That's the case whether we're doing regression binary classification or multi class classification.
So then what's the difference between these three cases.
The difference is that the meaning is assigned by us the designer of the program and we assign that
meaning through the loss function.
In other words it's up to us to come up with the appropriate lost function such that the meaning of
these two outputs is the Gaussian mean and variance
but you should be suspicious at this point.
A neural network layer can output any value but don't we have a constraint on the value of a variance.
In fact we require that a variance is positive in order to enforce the variance to be positive.
We can make the assumption that the none that work doesn't output the variance directly but rather the
log of the variance if you think hard this is just like the classification case.
The notion that work doesn't output probabilities directly.
Instead it outputs logics which are related to log probabilities.
In fact that has more constraints because not only do probability distributions have to have positive
values they also have to sum to one
using this information we can determine what the laws function should look like with respect to the
neuron that works outputs.
First let's assume that instead of having a neuron that work with two output nodes we simply have two
neural networks in our model and then the fourth function returns the output from these individual and
their own that works then the output of our model from the forward function will be a tuple with the
first element is the mean predictions and the second element is the log variance predictions.
Now we get back to our original question which was how do we create a custom lost function and Pi talk.
As always it's easiest to start with what we're familiar with the mean squared error.
If we had a regular road regression neural network that only outputs the mean prediction then here's
how you would do it.
The trick is there is no trick.
The lost function is merely a regular Python function that takes in any arguments you want.
Obviously some of them have to be pi to which variables for us this means the predictions and the targets
then inside this function we find the element y's difference between the predictions and the targets
square them and then take the mean.
Pretty straightforward so a custom lost function is just a regular function that does mathematical operations
using pi to urge.
So how do we do this.
When it comes to the full negative log likelihood First we still take in the same arguments the predictions
and the targets.
This time the predictions are tuples so we can split them up.
The first element becomes the mean which I'll come to and the second element becomes the log variance.
Since that XP function is the inverse of the log function I can take that e XP of the log to get the
variance itself which I'll call V then we simply need to plug and chug into our formula for the negative
log likelihood.
This consists of the coefficient term and the exponent term since we're in log space.
We can just add these two together and this gives us the element y's negative log likelihood after which
we can take the mean
so at this point you may be wondering how is a model like this useful.
And naturally you can apply this technique to any problem that contains uncertainty in the target function
if we recall the bias variance tradeoff.
This is the statement that the error of your model is the bias squared plus the variance plus an irreducible
error the irreducible error is called that because it's a characteristic of the data itself and your
model can't make a perfect prediction because of it.
This arises naturally.
For example suppose you're measuring the relationship between how long students study on a test and
the grade let's say you have one student who study 10 hours and got a ninety five percent.
And you have another student who study 10 hours and got an 85 percent.
In this case the input is the same but the target is different.
And since your model is only a function of the inputs it can't make a perfect prediction.
If there are two different possible target values for that single input as you can imagine this kind
of noise can appear everywhere.
So instead of ending with a generic statement like my model has a lower square you can quantify the
uncertainty of your model instead
when the variability of the target.
With respect to the input is not constant.
We say that it's hetero scholastic we can think of other situations where hetero skit activity city
can arise.
One popular application is with stock returns and stock prices.
These are well known to be hetero scholastic in finance.
We often want to avoid high variance because we think of variance as risk.
And so in order to minimize risk we should minimize variance.
And in order to minimize variance it would be useful to model not just the prices of the returns but
the variance itself.