In this lecture, we are going to answer the question, how does a model learn?
Let's start with linear regression and its most basic form linear regression just means line of best
fit.
As you may recall from your high school math studies, a line has the equation Y equals m x plus b here.
X is the input variable, M is the slope and B is the Y intercept.
When we put these together, we get the equation for a line.
And our job, of course, is to find a line that best fits our input data.
So you can imagine that our input data is a bunch of data points that approximately create the shape
of a line.
This plot is called a scatterplot.
So for each point X and Y in our data set, we're going to draw a dot on this chart.
Our goal is to find the line that best passes through all these data points.
In practice, this means finding the slope in y intercept.
One method you may have used in high school is to take a ruler and to try and draw the best line on
paper using visual inspection.
Of course, since this is data science, we no longer use such methods.
We have to be more systematic.
The line that you draw might not be the same as the line that I draw.
And furthermore, if we are in multiple dimensions, this won't be a line at all.
So it's clear that we need something better and more quantitative.
The idea is we're going to define an era function for each data point, which is a pair made up of the
input exi and the target why we're going to calculate a prediction, why had-I equals to exi plus b,
then we are going to take the squared difference between why and why had-I.
We're going to do this for each of our data points for I equals one up to end.
Once we've done this, we can add them all up and divide by end.
This is called the mean squared error.
It's the average square deviation between our predictions and our targets.
Basically, this is going to tell us how accurate our model is.
If our predictions are equal to the targets, then this era will be zero, because why I would be equal
to why had I then the more wrong we are, the larger this era becomes.
So that's generally how an error function should work.
As a side note, since some students get confused by this, at first, we have multiple names for this
error and we usually use them interchangeably.
Sometimes we call it a loss.
Sometimes we call it an objective or sometimes we call it a cost.
However, these all mean the same thing.
I like the term cost because it makes intuitive sense for business people as any good businessman would
do.
Your job is to minimize your cost.
And in fact, this statement alone tells us how to find the ways W.
So how do we minimize this cost or, in other words, make it as small as possible?
Well, it's time to turn it to our old friend calculus.
If you recall, we can find the minimum or a maximum of a function by finding where it's derivative
is equal to zero.
If you're not convinced by this, try to draw a curve which has a minimum or a maximum and draw the
slope at the minimum or maximum point.
If you drew your line correctly, it should be a horizontal line, meaning that the slope at this point
is zero.
And remember, the slope at each point is just the derivative.
So that's why we want to find the derivative.
Set it to zero and then solve for the parameter in question.
Now, usually we have more than one parameter, even in one dimensional linear regression.
This is the case because we have both the slope and the Y intercept.
So what happens when we have a function of more than one variable?
In this case, the derivative is actually called the gradient.
If you recall from your calculus studies, the gradient is just a vector or tensor of partial derivatives
for each of the variables, we're taking the gradient with respect to.
So, for example, if you take the square function in one dimension, that's called a parabola.
But if you have two input dimensions, it's called a parabola.
Beyond that, we can't picture it, but the basic idea is still the same.
Find the gradient, set it to zero.
So for the parameters, by the way, the term parameter is just another name for the weights.
So that's the W and B we've been referring to throughout this section.
Luckily, in this course, we won't be calculating any derivatives manually since I have many other
courses which do that a lot and by a lot, I really mean a lot.
So in this course, we're kind of excused from doing that because TensorFlow already does that for us
using a process called automatic differentiation.
Luckily, you don't have to know how that works either, because as its name suggests, it's automatic.
So the only thing you really do have to know is that a TensorFlow is going to automatically find the
gradient of all your weights and be TensorFlow uses these gradients to train your model.
At this point, you might be wondering if all I have to do is find the gradient and set it to zero.
Then why in our previous code did it involve an iterative training process?
Recall that we had to specify the number of epochs to train for.
And this resulted in a plot of loss per iteration, which we could check to confirm that the training
algorithm converges nicely.
Well, in actuality, it's not possible to actually solve for the equation that you get when you set
the gradient to zero.
Most of the time, the one exception to this in this course, at least, is linear regression where
we can solve it.
We call the solution an analytical solution or a closed form solution.
Basically, this means that we can express the optimal value of WNDB using an equation for logistic
regression and the rest of the models will be discussing.
It's not possible to do this.
Therefore, we need another approach.
That approach is called gradient descent.
Now, there's a lot more to gradient descent than what's in this lecture.
And you can check out the in-depth section of this course for that if you're interested.
For now, let's just talk about the basics.
Essentially, what we do is we start at a randomly initialized point for both WMD since this is random.
These probably don't lead to a small cost.
So then we find the gradient of our loss with respect to WMD.
We then take small steps in this direction to update W and B on each iteration.
Remember, these iterations are called epochs.
Mathematically, you can prove that this leads to a decrease in costs, although we won't discuss that
in this lecture.
You can imagine that this is exactly what goes inside the Carrera's Fit function.
This is really all that's happening for epoch and range epochs and then set to equal to W minus eight
times the gradient of J.
With respect to W and set B to B minus eight times the gradient of J with respect to B.
So again, all we're doing is taking small steps in the direction of the gradient with respect to W
and B, and one obvious question that arises from this is how small should these small steps actually
be?
Well, the step size is specified by this Greek letter ADA, which we call the learning rate.
This specifies how fast or slow we want to train our model.
It's important to set this value right, because if you don't, then your model won't get good results
even if your model is actually a good model.
Unfortunately, there's no direct method of choosing a good learning rate.
Generally speaking, the learning rate is something we call a hyper parameter.
This is to differentiate it from WMP, which are just regular old parameters.
It's called a hyper parameter because it's still a parameter, but it's not a parameter of the model
itself.
It's sort of like a meta parameter.
And in fact, no hyper parameters are really chosen directly.
It's more of a process of trial and error, along with intuition that you gain from practicing a lot
and seeing a lot of different examples.
As mentioned before.
One way to know if your lending rate is too high or too low is to check the loss per iteration after
training.
This is somewhat unfortunate because training can take a long time and you only know the result once
it's done.
If you use a bad learning rate and got bad results while you still had to wait for them, I really like
this visualization because it encapsulates how to choose the lending rate quite well.
If you're lending rate is too high, then your loss will just shoot off to infinity.
Intuitively, this is because you're overshooting the minimum and just ending up on the other side of
the canyon.
That means your steps are too large and you need to make them smaller.
Sometimes your lending rate might be a bit too high, so you'll see some convergence, but it'll converge
to a sub optimal value.
That's kind of what happened in the more example without learning rates scheduling.
If you're lending rate is too low, then you'll see a really shallow curve.
This is also not good because as I mentioned, training takes time.
So if your lending rate is too low, you'll have to wait longer.
But not only that, sometimes you can just get stuck at a suboptimal point.
That's also not good.
A good learning rate is in between these two extremes, and so what you'll probably end up doing is
finding the limits of your particular data set.
What's too high, what's too low and then trying different numbers in between.
Until you find something good.
Normally, we try numbers that are powers of 10.
For example, zero point one zero point zero one zero point zero zero one and so forth.