-: Hello and welcome back to the course on deep learning.
In today's tutorial, we're talking about gradient descent.
What we learned previously was that in order
for a neural network to learn,
what needs to happen is back propagation.
And that is when the error, the difference,
or the sum of square differences between Y hat
and Y is back propagated through the neural network
and the weights are adjusted accordingly.
So we saw that and today we're going to
learn exactly how these weights are adjusted.
So let's have a look.
This is our very simple version of neural ledge, perception
or a single layer feed-forward neural network.
And what we can see here is this whole process
in action where we've got some input value, then
we've got a weight, then a activation function is applied.
We have, we get Y hat
and then we compare it to the actual value.
We calculate the cost function.
So how can we minimize the cost function?
What can we do about it?
Well, one approach to do it is a brute force
approach where we just take all, lots
of different possible weights and look
at them and see which one works best.
And what we do is, for instance, we'd try out for, let's say
for example, a thousand weights and we'd try them out.
We'd get something like this for the cost function.
And this is a chart of
on the Y axis you have cost function on the vertical axis.
On the horizontal axis you have Y hat.
And because you can see the formula is Y hat minus
y squared, this is what the cost function would look
something like that.
And basically you'd find the best one is over here.
So very simple, very intuitive approach.
Why not do this brute force method?
Why not just try out a thousand different,
cost, thousand different parameters
or inputs for weights and see which one works the best?
You'll find the best one that way.
Well, if you have just one way to optimize, this might work.
But as you increase the number of weights,
increase the number of synapses in your network
you have to face the curse of dimensionality.
And so what is the curse of dimensionality?
The best way to describe this
or explain it is to just look at a practical example.
So remember this example we had when we were talking
about how neural networks actually work where
we were building or
running a neural network for proper evaluation.
So this is what it looked
like when it was trained up already.
Well, when it's not trained, before it's trained,
before we know which, what are the weights,
the actual neural network looks like this, right?
Because we have all these different possible synapses
and we still have to train up the weights.
And here we have a total of 25 weights.
So four times five at the start, plus five more
from the hidden layer to the output layer, 25 weights total.
And let's see how we could possibly brute force 25 weights.
It's a very simple neural network right here.
Very simple, just one hidden layer.
And how could we brute force outweigh
through a neural network of this size?
Well, let's do some simple mathematical calculations.
We have 25 weights.
So that means
if we have a thousand combinations that we're gonna test
out for every weight, the total number
of combinations is a thousand
to the power of 25 or a thousand,
or 10 to the power of 75 different combinations.
Now, let's see how Sunway TaihuLight,
the world's fastest supercomputer
as of June 2016.
What, how would it approach this problem, right?
So Sunway TaihuLight looks
like this, is a whole huge building pretty much
for this one supercomputer.
And it got the Guinness World Record
for being the fastest supercomputer.
Right now, it is the fastest supercomputer in the world
and Sunway TaihuLight can operate
at a speed of 93 petaflops.
FLOPS stands for floating operation per second.
So it can do 93 to the power times 10
to the power of 15 floating operations per second.
That's how quick it is.
In comparison, average computers right now
they do like just over several gigaflops and so on.
So it like kind of those ranges,
way less than Sunway TaihuLight.
So Sunway TaihuLight is in the forefront of technology
and let's say hypothetically that it can do one, test
one combination for our neural network in one FLOP,
basically in one floating operation.
That is not possible.
That is not practical because you need multiple floating
operations to test out a single weight in your neural.
But even let's, let's give it a head start.
Let's say that it can do that.
In a ideal world, it can do that in one floating operation.
It can do one test per one floating operation.
That means it will still require 10 to the power
of 75 divided by 93 times 10 to power of 15 seconds to
come to, run all of those tests to brute force
through that network.
So that means one or approximately 10 to the power
of 58 seconds, and that is the same as 10 to the power
of 50 years.
That is a huge number,
that is longer than the universe has existed.
And that is definitely not going to simply
this number is so huge, it's just definitely
not going to work for us at all in our optimization.
So there we go.
This is a no-no, even on the world's fastest
supercomputer Sunway TaihuLight.
So we have to come up with a different approach.
How we going to find the optimal weight?
By the way, this, our neural network was very simple.
What about if the neural networks looks
like something like this, right?
Or even greater than that, then yeah
it's just not going to happen at all, ever.
So the method we're going to be looking
at is called gradient descent.
And you may have heard of it already.
If not, we will find out what it is right now.
So there's our cost function
and now we're going to see how we can faster, come up
with a faster way to find the best option.
So let's say we start somewhere.
You gotta start somewhere.
So we start over there and from that point in the top left,
what we're going to do is we're going to look
at the angle of our cost function at that point.
So we're just going to basically, that's why
it's called gradient, because you have to differentiate.
We're not going to look at the mathematical equations.
We will provide some tips
on additional reading at the end of the next lecture.
But basically you just need to differentiate, find
out what the slope is in that specific point
and find out if the slope is positive or negative.
If the, if the slope is negative
like in this case, means that it, you're going downhill.
So to the right is downhill, to the left is uphill.
And from there it means you need to go right.
Basically you need to go downhill.
And that's what we're going to do.
Vroom, takes a step, right?
The ball rolls down.
Again, same thing.
You calculate the slope.
This time the slope is positive, meaning right is uphill
left is downhill and you need to go left.
And you roll the ball down and again
you calculate the slope and you roll the ball right.
There we go.
So that's how you find in, in simple terms
that's how you find the best weights,
the best situation that minimizes your cost function.
Of course, it's not going to be like a ball rolling,
it's going to be a very zigzaggy type of approach
but it's easier to remember or kind of it's--
it's more fun to look at it as a ball rolling.
But in reality, yes, you just
it is going to be like a step-by-step approach.
So it's gonna be a zigzaggy type of method.
Yeah, and also there's, there's lots of
other elements to it.
There's things like, for instance, why,
like why does it go down?
Why does it not like go away over the line?
So it could have jumped out of this, gone upwards instead
of downwards and things like that.
So there are parameters that you can tweak, and again
we will mention where you can find out more on that.
And plus we'll have this in the practical application.
But in the simplest intuitive approach,
this is what is happening.
We are getting to the bottom
by just understanding which way we need to go.
Instead of brute-forcing through thousands and thousands
and millions and billions and quadrillions of combinations,
we can just simply every time have a look
at where is, where, which way is it sloping?
So right like you are, you imagine you're standing
on a hill, which way does it feel that it's going downwards?
And whichever way it is going downwards,
you just keep walking that way you like
you take 50 steps that way
and then you, you assess again, okay
which way is it going downwards this way?
Okay, now take 50 steps or less, take 40 steps that way
so it gets less and less and less as you get closer.
So there, here's an example
of gradient descent applied in a two-dimensional space.
So that was a one-dimensional example.
Here we have a two-dimensional space
for the gradient descent.
As you can see, it's getting closer
to the minimum and it's also called gradient descent
because you're descending
into the minimum of the cost function.
And finally,
here's the gradient descent applied in three dimensions.
This is what it looks
like if you project it onto two dimensions
you can see it's zigzagging its way into the minimum.
So there we go.
That was gradient descent.
In next tutorial we'll talk
about stochastic gradient descent is will be a continuation
of this tutorial and I look forward to seeing you there.
Until next time, enjoy deep learning.