Instructor: Hello and welcome back
to the course on Deep Learning.
Today we're going to wrap up with back propagation.
All right, so we already know pretty much everything we need
to know about what happens in the neural network.
We know that there's a process called,
forward propagation where information is entered
into the input layer and then it's propagated forward
to get our y hats, our output values,
and then we compare those to the actual values that we have
in our training set and then we calculate the errors.
Then the errors are back propagated through the network
in the opposite direction,
and that allows us to train the network
by adjusting the weights.
So, the one key important thing to remember here is
that back propagation is an advanced algorithm driven
by very interesting
and sophisticated mathematics, which allows us
to adjust the weights, all of them at the same time.
All of the weights are adjusted simultaneously.
So, if we were doing this manually or
if we were coming up with a different type
of algorithm, then even if we calculated the error
and then we were trying to understand what effect each
of the weights has on the error, we would have
to somehow adjust each of the weights independently
or individually.
The huge advantage of back propagation
and this a key thing to remember is
that during the process of back propagation simply
because of the way the algorithm is structured, you are able
to adjust all of the weights at the same time.
So, you basically know which part of the error each
of your weights in the neural network is responsible for.
Now, that is the key fundamental underlying principle
of a back propagation and this was why it picked up
so rapidly in the 1980s and this was a major breakthrough,
and if you'd like to learn more about that
and how exactly the mathematics works
in the background, then a good article,
which we've already mentioned, is the,
"Neural Networks and Deep Learning" is actually a book
by Michael Nielsen.
There you'll find the mathematics written out
and it'll help you understand how exactly this is possible.
But for now, for our purposes, if from an intuition point
of view, the important part is to remember that
that's what back propagation does.
It adjusts all of the weights at the same time.
And now we're going to just wrap everything up
with a step-by-step walkthrough of what happens
in the training of a neural network.
All right, so step one, we randomly initialize the weights
to small numbers, close to zero, but not zero.
We didn't really focus on the initialization
of weights during the initialization tutorials
but the weights have to start somewhere
and they're initialized with random values near zero
and from there, through the process
of forward propagation back propagation,
these weights are adjusted until the error is minimized,
until the cost function is minimized.
Then step two, input the first observation
of your data sets to the first row into the input layer.
Each feature is one input node.
So basically, take the columns
and put them into the input nodes.
Step three, forward propagation
from left to right, the neurons are activated in a way
that the impact of each neurons activation is limited
by the weights.
So the weights, basically,
determine how important each neurons activation is.
Then propagates the activations,
until getting the predicted result, y hats in this case.
So basically, you propagate from left to right,
or you go all the way until you get to the end
and you get your y hat.
Then compare the predicted result
to the actual result, measure the generated error.
And then you do the back propagation from right to left.
The error is back propagated, update the weights according
to how much they're responsible for the error.
Again, you are able to calculate that because
of the way the back propagation algorithm is structured.
The learning rate decides
by how much we update the weights learning rate,
it's parameter you can control in your neural network.
Step six, repeat steps one to five and update the weights
after each observation.
That is called reinforcement learning, and in our case
that was stochastic gradient descent
or repeat steps one to five
but update weights only after a batch of observation.
So batch learning, it's either full gradient descent
or batch gradient descent or mini batch gradient descent.
And step seven, when the whole training set passed
through the artificial neural network
that makes an epoch redo more epochs.
So basically you just keep doing that
and doing that and doing that and
to allowing your neural network to train better
and better and better, and constantly adjust itself,
as you minimize the cost function.
So there we go, those are the steps you need to take
to build your artificial neural networks and train it.
And these are the steps that you will be taking together
with Hadelin in the practical tutorials.
Wish you the best of luck
and I look forward to seeing you next time.
Until then, enjoy deep learning.