In this lecture we are going to answer the question how does a model learn.
Let's start with linear regression and its most basic form linear regression just means line of best
fit.
As you may recall from your high school math studies a line has the equation y equals M X plus b here
X is the input variable M is the slope and B is the y intercept.
When we put these together we get the equation for a line and our job of course is to find a line that
best fits our input data
so you can imagine that our input data is a bunch of data points that approximately create the shape
of a line.
This plot is called a scatter plot.
So for each point x and y in our dataset we're going to draw a dot on this chart.
Our goal is to find the line that best passes through all these data points.
In practice this means finding the slope and y intercept one method.
You may have used in high school is to take a ruler and to try and draw the best line on paper using
visual inspection.
Of course since this is data science we no longer use such methods.
We have to be more systematic.
The line that you draw might not be the same as the line that I draw.
And furthermore if we are in multiple dimensions this won't be a line at all.
So it's clear that we need something better and more quantitative.
The idea is we're going to define an error function for each data point which is a pair made up of the
input x eye and the target y.
We're going to calculate a prediction y had I equals to m exile plus B then we are going to take the
squared difference between Y and Y had I we're going to do this for each of our data points for I equals
one up to n once we've done this we can add them all up and divide by n.
This is called the mean squared error.
It's the average squared deviation between our predictions and our targets.
Basically this is going to tell us how accurate our model is.
If our predictions are equal to the targets then this area will be zero because why I would be equal
to I had.
Then the more wrong we are the larger this error becomes.
So that's generally how an error function should work
as a side note.
Since some students get confused by this at first we have multiple names for this error and we usually
use them interchangeably.
Sometimes we call it a loss.
Sometimes we call it an objective or sometimes we call it a cost.
However these all mean the same thing.
I like the term cost because it makes intuitive sense for business people as any good businessmen would
do.
Your job is to minimize your cost and in fact this statement alone tells us how to find the ways W.
So how do we minimize this cost or in other words make it as small as possible.
Well it's time to turn to our old friend calculus if you recall we can find the minimum or a maximum
of a function by finding where its derivative is equal to zero.
If you're not convinced by this try to draw a curve which has a minimum or maximum and draw the slope
at the minimum or maximum point.
If you drew your line correctly it should be a horizontal line meaning that the slope at this point
is zero.
And remember the slope at each point is just the derivative.
So that's why we want to find the derivative set it to zero and then solve for the parameter in question
now usually we have more than one parameter even in one dimensional linear regression.
This is the case because we have both the slope and the y intercept.
So what happens when we have a function of more than one variable in this case.
The derivative is actually called the gradient.
If you recall from your calculus studies the gradient is just a vector or tensor of partial derivatives
for each of the variables we're taking the gradient with respect to so for example if you take the square
function in one dimension that's called a parabola but if you have two input dimensions it's called
a parabola void.
Beyond that we can't picture it but the basic idea is still the same find the gradient set it to 0 solve
for the parameters by the way the term parameter is just another name for the weights.
So that's the W. and b we've been referring to throughout this section
luckily in this course we won't be calculating any derivatives manually since I have many other courses
which do that a lot.
And by a lot I really mean a lot.
So in this course we're kind of excused from doing that because pi torch already does that for us using
a process called automatic differentiation luckily you don't have to know how that works either because
as its name suggests it's automatic.
So the only thing you really do have to know is that a PI torch is going to automatically find the gradient
of all your weights and be pi which uses these gradients to train your model
at this point you might be wondering if all I have to do is find the gradient and set it to zero.
Then why in our previous code did it involve an iterative training process.
Recall that we had to specify the number of epochs to train for and this resulted in a plot of loss
per iteration which we could check to confirm that the training algorithm converged nicely.
Well in actuality it's not possible to actually solve for the equation that you get when you set the
gradient to zero.
Most of the time the one exception to this in this course at least is linear regression where we can
solve it we call the solution an analytical solution or a closed form solution.
Basically this means that we can express the optimal value of W and B using an equation for logistic
regression and the rest of the models we'll be discussing.
It's not possible to do this.
Therefore we need another approach.
That approach is called gradient descent.
Now there's a lot more to gradient descent than what's in this lecture.
And you can check out the in-depth section of this course for that if you're interested.
For now let's just talk about the basics.
Essentially what we do is we start at a randomly initialize point for both WMD.
Since this is random these probably don't lead to a small cost.
So then we find the gradient of our loss with respect to W and b we then take small steps in this direction
to update W and B on each iteration.
Remember these iterations are called epochs mathematically.
You can prove that this leads to a decrease in cost although we won't discuss that in this lecture.
You can imagine that this is exactly what goes on inside the optimizer that step function.
This is really all that's happening for epoch in range epochs and then set w equal to W minus eight
times the gradient of J.
With respect to W and set B to be minus eight times the gradient of J with respect to B.
So again all we're doing is taking small steps in the direction of the gradient with respect to W and
B and one obvious question that arises from this is how small should these small steps actually be.
Well the step size is specified by this Greek letter ADA which we call the learning rate that specifies
how fast or slow we want to train our model.
It's important to set this value right because if you don't then your model won't get good results even
if your model is actually a good model
unfortunately there is no direct method of choosing a good learning rate.
Generally speaking the learning rate is something we call a hyper parameter.
This is to differentiate it from W and B which are just regular old parameters.
It's called a hyper parameter because it's still a parameter but it's not a parameter of the model itself.
It's sort of like a meta parameter and in fact no hyper parameters are really chosen directly.
It's more of a process of trial and error along with intuition that you gain from practicing a lot and
seeing a lot of different examples
as mentioned before.
One way to know if your learning rate is too high or too low is to check the last per iteration after
training.
This is somewhat unfortunate because training can take a long time and you only know the result once
it's done.
If you use a bad learning rate and got bad results while you still had to wait for them I really like
this visualization because it encapsulates how to choose the lending rate quite well.
If you're learning rate is too high then your loss will just shoot off to infinity.
Intuitively this is because you're overshooting the minimum and just ending up on the other side of
the canyon that means your steps are too large and you need to make them smaller.
Sometimes your lending rate might be a bit too high so you'll see some convergence but it'll converge
to a sub optimal value if you're lending rate is too low then you'll see a really shallow curve.
This is also not good because as I mentioned in training takes time.
So if your lending rate is too low you'll have to wait longer.
But not only that sometimes you can just get stuck at a suboptimal point.
That's also not good.
A good learning rate is in between these two extremes and so what you'll probably end up doing is finding
the limits of your particular dataset what's too high what's too low and then trying different numbers
in between until you find something good.
Normally we try numbers that are powers of 10 for example zero point one zero point zero one zero point
zero zero one and so forth.