We've just looked at error sanctioning, and that has permitted us to see how this model A um, with
parameters M and C different from this other model B with its own parameters m and C.
Let's say, for example, given that we have um, this line y so m is say for example zero and c three.
And then this other m could be two and then c could be let's say four.
With the model B having a higher loss or having a higher error as compared to the model A.
Now the whole idea of um training optimization is simple.
What we want to do is take a model which has been initialized.
Let's take a model whose parameters have been initialized to some random values, like for example,
zero three or even zero, zero or whatever random values, and then update those parameters such that
we get a better performing model like this one here with parameters two four.
So let's suppose that we randomly initialize our model, um, for to have m equals zero and c equal
three.
Then we want to update this such that zero tends to something close to two, and three turns to something
close to four.
Now, to carry out this transformation, we shall make use of um, an algorithm known as the stochastic
gradient descent.
This algorithm is quite simple.
Suppose we have a weight.
This weight could be m or c.
Thus we have the a parameter.
Uh, let's call this parameter.
Or let's say let's say this initial value.
We are going to subtract.
We're going to take this weight minus, uh, learning rate.
Times the partial derivative of the loss.
This loss is the same as the error we had computed, uh, with respect to that particular weight.
So let's have this.
This is minus all this here.
And when you have this, I.
So this initial width or the initial weight value um is here.
So we have w I minus learning rate times the partial derivative of l the loss with respect to I.
And then this now gives us w f.
That's this final value.
So we could go from um if we pick out some let's pick out some values.
Let's suppose that uh our w initially is zero minus, um, let's say a learning rate of um, 0.1.
Times.
Um, a partial derivative of, let's say -20 will now give us a w f, which is two.
And that's how we could move from, um, this initial w to this two.
But it should be noted that in practice we don't generally just move from, uh, very poorly performing
model to one that is very well performing in a single step.
So we are going to go through this same algorithm in many steps.
So we could start with let's get back.
We could start with um, this partial derivative of um let's say even five.
So here, here we have 0.1 times ten.
Uh, times one times ten is actually uh, one divided by two.
That's 0.5.
So we could go from 0 to 0.5.
You see we go from 0 to 0.5.
And then in our next step, because now our W has turned to 0.5.
That's from zero.
We went from 0 to 0.5.
We will replace this zero here with 0.5.
So instead of having zero we would have now 0.5.
The learning rate is still the same.
Um, although it could change as we keep training.
And then this partial derivative, we could say for example that is ten.
So let's um, this is 0.5.
Let's take this off here and then replace this with a negative ten.
So here we have negative ten.
Now you see we have one.
So we have 0.5 minus or rather 0.5 plus one.
So 0.5 plus one gives us um 1.5.
So let's take this off again.
So instead of 0.5 now we have 1.5.
Take this off.
And we have now 1.5.
So you see how we go from zero.
We go from 0 to 0.5.
And then 21.5.
And the reason why I would expect to have uh, better values of w better in this case is values of w
such that the, the straight line would be close to enough to this point as compared to this other poorly
performing model.
And so though, as we're saying, the reason why we are sure that we are going to keep getting towards
this, um, better performing values of W is because we are including the loss in this, um, algorithm.
So when we calculate our loss, we look for this partial derivative with respect to the weights.
And is this partial derivative that permits us to adjust the weights.
Now this learning rate helps us adjust by how much we are going to, um, modify the weights.
So if we had a large learning rate here, or let's say if we had a larger learning rate, not large
because learning rates are relative.
So if we had a larger learning rate of um instead of 0.1 if we had one.
So let's take this off.
If we had one, you would find that right from the first step we would have um, five.
So we would, we would just jump from 0 to 5.
So this controls how much we, we improve our model.
Now, if you have no background in calculus, you shouldn't bother.
As with TensorFlow, all we need to do to make use of this algorithm is by just specifying that our
optimizer is SGD.
So you just need to put in the string and you're good to go.
You don't really need to write all this code.
So getting back um here again we have let's let's plot the loss against the weights.
So we have a plot of loss against weights.
Now our loss here is actually something like y minus y prime.
All of that square is true.
We have a sum but we're going to ignore that.
So it's going to look like it's going to look something like this.
You see this quadratic function.
And even if you have no background in calculus, you should just understand that when we talk about,
uh, partial derivative or derivative in general, uh, and we have a curve that has to do with the
slope of that curve.
So if you, if you have at this point, if at this point you have w and you have the loss at this point
the, the slope, you have a slope which is, which looks like this.
If you draw a tangent and you calculate a slope, you try to calculate the slope.
You find that here the slope is going to be higher than if we draw a tangent at this point.
If you look at this point here and you draw this tangent well let's get back and try to draw a tangent.
And we have a slope.
The slope is smaller as compared to this.
So if this is let's call this slope a and let's call this slope b.
So slope a is greater than slope b.
And so what this means is the partial derivative of the loss with respect to the weight at this point
is greater than the partial derivative of the loss with respect to the weight at this other point.
Now let's see how this, uh, plays when we when it comes to updating the weights, remember that the
ideal situation is that our loss should be zero.
So what we want to do here is minimize the loss.
So, um, if we get to a point like this, then that's fine because we've gotten to the smallest possible
value of our loss.
And what this means is you may even have two and four and, uh, as weight as, um, M and C as the
parameters, but the, the loss will be maybe somewhere about, uh, around here.
And so there might still be room for improvement.
And so, uh, what happens is we are going to take this weight.
You see, we have the weight, subtract the take the subtract the learning rate times this slope actually.
So when we say partial derivative of the loss with respect to the weight, what we actually are doing
here is we're subtracting that slope.
And so if we are having, um, this weight, we imagine we have this weight here and that this, this
weight links leads us to this point here.
There we go.
We have this point here where our loss is, is quite high, which is not what we desire.
Then you would find that the slope, the slope is much larger.
Just like with this slope here, the slope is much larger as compared to if we have a weight value around
here where we have a smaller slope, and because the slope is much larger, you would find that, um,
multiplying this larger slope by the learning rate would lead to a larger change in the weight.
So, um, this larger change in the weight will tend to make the weight, um, or this weight value
move to the left.
Because now we're taking all the weight minus a large value.
And so we'll tend to move now to the left.
Now if we're in this other direction, if we come in this other direction, that if we have a weight
value this other direction, then, um, taking w minus this learning rate times the slope, because
the slope is now negative, we have a negative slope.
We'll have w minus a negative.
That's w plus the learning rate times the absolute value of the slope.
And so now we'll be moving towards the right.
And so you see how graphically the stochastic gradient descent algorithm permits us to move the weight
towards values which will permit us have, um, a smaller loss.
Another interesting point to note is the fact that as we as our weight approaches this values where
the corresponding loss is very small, the slope also becomes very small.
And so the change in weight isn't that large.
So so if we have if we get to a point like this here, if we get to this point, let's say we get to
this point here, which is very close to the optimal weight value, where we're going to have the smallest
possible loss, then this slope here is going to be very small.
And so when you do w minus learning rate times that very small slope, you will find that the value
of w will change just a little bit.
And so what this tells us is when we are far away from that um optimal weight value, the slope is large.
And then when we approach that, uh, optimal weight value, the slope is small.
And so what happens generally during training is as we start training, the loss value, the loss the
loss value drops much faster.
And as we keep getting much better weight values, this loss value continues dropping but much more
slowly.
And this is what we call um, model convergence.
Now the choice of this learning rate is very important because if our learning rate is too big, then,
um, multiplying it by a large slope will take this weight to, uh, this other side, which is not
what we want.
And if it's too small, then we'll even if we have a large slope, the change in the weight is going
to be also too small.
And so the learning process is going to be too slow.
So we want to choose, uh, an adequate value for the learning rate so that our training goes fast and
smooth.
Diving back to the code, you see all we need to do, um, is just specify SGD and we're going to have
the the w minus learning rate times the partial derivative with respect to the learning with respect
to the weights, um, which are going to be calculated um, under the hood.
So that's it.
Let's get back and print out the shape that's y and x and y.
And then we could start with the training.
So here we just need to do model dot fit.
And then we specify x.
We specify y we give number of epochs.
Remember we said we are not going to start updating these parameters and then dive.
Or after one step just get straight to the optimal values.
Generally we go through several steps and each one of these step is called an epoch.
So we'll specify a number of epochs to say um ten.
And then we'll variables we say one.
So let's run that and see what we get.
Um has no this should be epochs on that.
And as you could see, the model, um, has trained for ten epochs.
Now you'll notice how this loss value keeps dropping.
See, we go from 308 or 308,519.
We drop um to 308,516.
If you increase the number of epochs, you would find that these values are going to drop even much
more, um, as compared to what we've seen already.
So you see, keeps dropping.
Now we get to 308, um, 485.
Now, although we specify this optimizer to be SGD, that's stochastic gradient descent.
Um, we could also have other optimizers.
So you could check out here.
But most of these are just flavors of the SGD, like the Adam and Adam W, which are the one of the
most popular optimizers we have today.
So click on this.
Adam.
You see we have Adam Optimizer, which as we said is a flavor of the SGD and it has now many more parameters.
You could feel free to check out all this in the documentation.
So let's just let's just copy this out or let's just get back to the code and import.
Um, Adam.
So we have get back to the top we have from TensorFlow.
TensorFlow Keras, Keras optimizers.
We're going to import um Adam.
So that's it.
You could import Adam.
You can import Adam um w or any other optimizer available in TensorFlow.
So that's it.
Once you import that you could now replace this SGD we had right here.
So instead of this now we just have Adam.
And then you could specify it's learning rate.
So we have learning rate by default it was given to be 0.001.
Yeah that was it.
Um take that off okay.
So by default we have this.
We also have the values for beta one, beta two and so on and so forth.
But let's just focus on the learning rate.
So by default we have this.
We could change this to say um one e negative five or some other value.
So let's just maintain the one e negative three.
And then just as before we compile the model.
And then we go ahead and train.
Now while training we did not keep the history of all these losses.
So let's say history.
Um there we go.
So we're still going to have the same output.
But now after the training is done, you could do history dot history, and you would see that you will
get all these different loss values.
So you get all the different loss values.
See we have this dictionary.
We have loss um, and we have all these values right up to the end.
So let's, let's now do history dot history and then we get the loss.
See we have directly all the different loss values from the epoch zero right up to the last epoch.
We can now plot out this loss values um using matplotlib.
So let's simply do, let's first of all import matplotlib.
We have from here um import matplotlib.pyplot as plt on that.
Well think of this.
Oh and that should be fine.
So we run that again.
That's fine.
We get back and then we, um, try to plot all this.
Uh, I'll try to plot out this loss.
So we see that, um, graphically.
There we go.
We have our matplotlib plot, um, history, history loss.
There we go.
We also want to have a title.
So here we have model model loss.
Then we want to have a y label.
Y label is simply the loss loss.
And then we have the X label.
The X label is the epoch um epoch.
And then we have um legend.
Legend.
We have train.
Well let's see train.
Okay, let's put on the list and we have train.
That's fine.
And now we do plot show.
There we go.
You can see our loss and you see that, um, it's actually dropping.
And if we compare this, uh, plot to what we had already in the board, we see that we're still very
far from the model convergence.
So we could still, uh, do better with this model, or we could still get, uh, much smaller loss
values.