-: Hello and welcome back to the course on deep learning.
Today we talk about stochastic gradient descent.
Previously, we learned about gradient descent
and we found out that,
it is a very efficient method to solve
our optimization problem where
we are trying to minimize the cost function.
It basically takes us from 10 to the power of 57 years
to solving a problem within minutes or hours
or within a day or so.
And it really helps speed things up
because we can see which way is downhill
and we can just go in that direction
and take steps and get to the minimum faster.
But the thing with gradient descent
is that this method requires
for the cost function to be convex.
And as you can see here,
we've specifically chosen a convex cost function.
Basically, convex means that the function looks similar
to what we are seeing now.
It's just convexed into one direction
and that it, in essence, has one global minimum
and that's the one that we're going to find.
But, what if our function is not convex?
What if our cost function is not correct?
What if it looks something like this?
Well, first of all, how could that happen?
Well, that could happen because if we,
first of all, choose a cost function
which is not the square difference between Y hat and Y
or if we do choose the cost function, which is like that
but then in a multidimensional space
it can actually turn into something that is not convex.
And so what would happen,
in this case if we just tried to apply
our normal gradient descent method,
something like this could happen.
We could find a local minimum
of the cost function rather than the global one.
So, this one was the best one and we found the wrong one.
And therefore we don't have the correct weights.
We don't have an optimized neural network.
We have a subpar neural network.
And so, what do we do in this case?
Well, the answer here is stochastic gradient descent.
And it turns out the stochastic gradient descent doesn't
require for the cost function to be convex.
So let's have a look at the two differences
between the normal gradient descent that we talked about
and the stochastic gradient descent.
So normal gradient descent is when we take all of our rows,
we plug them into our neural network, and once again,
here we've got the neural network copied over several times
but the rows are being plugged
into that same neural network every time.
So there's only one neural network.
This is just for visualization purposes.
And then once we've plugged them in,
we've calculated our cost function
based on the formula on the right
and looking at the chart at the bottom.
And then we adjust the weights then,
and this is called the gradient descent method
or it's also, the proper term
is the batch gradient descend method.
So, we take the whole batch from our sample,
we apply it, and then we run that.
The stochastic gradient descent method
is a bit different.
Here, we take the rows one by one.
So, we take this row, we run our neural network
and then we adjust the weights.
Then we move on to the second row.
We take the second row, we run our neural network,
we look at the cost function,
and then we adjust the weights again.
Then we take another row, take row three,
we run our neural network,
we look at the cost function, we adjust the weight.
So basically we are looking at,
we're adjusting the weights after every single row,
rather than doing everything together
and then adjusting the weights.
Two different approaches,
and now we're going to just compare the two side by side.
So, here they are.
This is how to visually remember them.
So, you've got the batch gradient descent
where you're adjusting the weights after you've run them,
after you've run all of the rows in your neural network.
And then, basically you adjust the weights
and you run the whole thing again,
iteration, iteration, iteration.
In the stochastic gradient descent method,
you run one row at a time,
and you adjust the weights
you adjust the weights, you adjust the weights,
and then you do everything again and again.
And that is called the stochastic gradient descent method.
The main two differences are,
that the stochastic gradient descend method
helps you avoid the problem
where you find those local extremos
or local minimums rather than the overall global minimum.
And the reason for that, in simple terms,
is that SGD or the stochastic gradient descent method,
has much higher fluctuations because it can afford them.
It's doing one iteration or one row at a time
and therefore the fluctuations are much higher
and it is much more likely to find
the global minimum rather than just the local minimum.
And the other thing about the stochastic gradient
descent method compared to the batch gradient
is that it's faster.
Like, the first impression that you might have is,
because it's doing every single row one at a time,
it is slower.
But actually, in fact, it is faster
because it doesn't have to load up all the data into memory,
and run, and wait until all of those rows
are run all together.
It can just run than one by one,
so it's a much lighter algorithm.
It's much faster in that sense.
So though, it's has way more, in those senses,
it has more advantages
over the batch gradient descent method.
The main advantage or the main pro
for the batch gradient descent method is that
it is a deterministic algorithm rather than
stochastic gradient descent, being a stochastic algorithm,
meaning it's random.
And with the batch gradient descent method,
as long as you have the same starting weights
for your neural network,
every time you run the batch gradient descent method,
you will get the same iterations,
the same results for the way your weights are being updated.
For stochastic gradient decent method,
you won't get that because it is a stochastic method.
You are picking your roles,
possibly at random and you are updating your neural network
in a stochastic manner.
And therefore you are just going to,
every single time you run
the stochastic gradient decent method,
even if you have the same weights at the start,
you're going to have a different process,
different iterations to get there.
So that's in a nutshell what stochastic gradient descent is.
Also, there's a method in between the two
called the mini batch gradient descent method,
where you combine the two and you basically run,
rather than running a whole batch or running one at a time,
you run batches of rows,
maybe 5, 10, 100, however many you decide to set,
you run that number of rows at a time,
then you update your weights,
and you update your weights, and so on.
And that's called the mini batch gradient descent method.
If you'd like to learn more about gradient descent,
there's a great article which you can have a look at.
It's called, "A Neural Network in 13 Lives of Python
(Part 2 - Gradient Descent)" by Andrew Trask.
And the link's below; it's on GitHub, 2015 article.
Very well-written, in very simple terms.
It's got some interesting philosophical,
or just interesting thoughts on
how to apply gradient descend whether,
you know, the advantages and disadvantages,
and how to do things in certain situations.
So, he's got some very cool tips, tricks, and hacks.
Very easy read, so definitely check that out.
And another one, a bit more heavier read,
for those of you who are into mathematics,
who want to get to the bottom of the mathematics,
why gradient descent in that specific,
what are the formulas that are driving gradient descent?
How is it calculated?
And so on.
Check out the article or actually the book.
It's a free online book called,
"Neural Networks and Deep Learning" by Michael Nielsen,
2015 book.
It's just basically, it's all online.
You can go ahead and check it out there.
And they're, again, very soft introduction
to the mathematics.
But the mathematics are pretty heavy as you go along,
as you read through the article.
But at the same time, it gets you into into that move-
I think even it has like a warm up chapter
where you first warm up with the math
and then you jump into them.
So, interested in math?
Then, this is the article to go to.
And, there it goes.
So, that's in a nutshell the difference
between gradient descent and stochastic gradient descent
and how the two work.
And on that note, we're going to wrap up today's tutorial.
I look forward to seeing you on the next one.
And until then, enjoy deep learning.