Hi and welcome to the section on back propagation.
This is what makes neural net trainable.
So let's take a look at this.
So as I just said, this is the most amazing algorithm that allows neural networks of convolutional
neural networks to be trained.
And what it does, it uses the loss to tell us how much to change or detail gradients so that we reduce
the overall loss.
So back propagation, this sounds like an ad for it, but basically this is what it is using loss to
make gradients better, but cheesy enough.
So anyway, here's an example of a very basic, simple example of the weights and biases and inputs
and outputs and formulas and how we get them for a very simple neural network.
Here we have one input tool sorry, two inputs, two hidden nodes and two output nodes.
So you can see these are the weights here represented when the S on the edges here of these connections
we have with W one point two, w two point three, five and so on and so on.
And then we have two biases here and nodes here, and we also have the inputs and the output from those
here.
So you can see output one is given by five times each one out plus w six plus eight times it's two out.
That's this one here and this one here, plus b two, which is the last set sets of biases.
So using the lost value back propagation can tell us now for the next iteration.
How much should we increase or decrease?
I should say W five to or the actually I do have.
I do say it here to reduce the overall loss.
So we just want to know if we have to change W five, do we go for small increase a big, big decrease?
How do we know?
Well, that's what we'll discuss with back propagation.
So no moving right to left back propagation updates to gradients node by node on a regular basis and
this is done for all nodes, show the network.
However, this magic just seems magical yet and right now, but it isn't explained to you yet, and
that's where we keep going.
So you can see how to move ahead with these arrows, and that's how the back propagation algorithm works
in theory.
Now let's look at the actual process behind it.
So firstly, remember, we followed propagate and input data and input image into the network, and
then we back propagate the to the lower weights to Lord loss.
But this simply tunes the words for that particular input.
We need to improve it to generalize to new data or unseen data.
And but to do that, we need to actually input all our training data into the neural network.
So you can imagine how many times we have two data gradients.
This is a continuous process, and basically this is the training process of the neural network.
So let's take a look at how we update OLEDs.
So what do we add ingredients to look like?
Again, let's go back to our simple neural network here, and let's take a look at this.
The output of hidden node one that's here.
The output is equal to AI one times W one as input one times one plus AI, two terms W2.
You can see the connections coming in plus b one.
That's the shared bias between these nodes and that's how we get the output of this node.
So just so you know that these hidden notes here we're talking about in a convolutional neural network
sense, there are two filters here.
However, it's a lot more confusing to understand when I'm using this example as opposed to this example.
So let's stick with this for now.
So how does it work?
Something called the general.
You may have heard this from your high school math.
Basically, if we have two functions, why f equals f of you and you Geof X, then the derivative of
Y is given by this.
Sorry, it's a bit too fast, so you can see it held.
It's all just slow this down.
You can see it.
Mathematically, you may get a refresher from the other derivatives in high school math.
The D y of a d y of the X is equal to divide.
You multiply by D you the ADX.
So that about a simple back propagation example.
All right.
We want to know how much changing five changes total error.
Well, that's what exactly exactly the partial derivatives can give.
This can tell us the change and error with respect to the change in way and w sorry.
So we just need to find us and to do to find this, we can actually just use the general going backward
to.
I don't have the formula here.
It's a bit long, but to cut to get the calculation for this.
So you have to use some of the error of the two outputs.
Just in case you didn't know, that's a total error u of T here.
So let's take a look at this.
The new W5 has a new wave of Note5 here is going to be equal to minus lambda multiplied by this change.
Remember that change here was how much we needed to change the total Arabi by changing WW here.
So you have me, you may have noticed we introduced a new parameter called lambda lambda is or a learning
rate, so it controls how big a jump we take.
So now that we know the value we need to take to update W5 so that we lower the loss, that's why it's
a negative sign here.
We need to know.
We need to know, adjust this factor that we're changing by, by the lambda factor.
Because if we change by big jumps, that's going to cause problems, which I'll explain to you in the
next section.
But big jumps basically cause the neural network to oscillate and never convert.
Yeah, lot.
We can learn a lot better, a lot faster, I should say.
However, they can be stuck in the local minimums and sometimes never really converge to the lowest
value, one to smoldering rates while training much more slowly.
They tend to converge to a much better global minimum.
So now let's stop there and we'll move on to gradient descent.
Thank you.