Now we want to understand back propagation.
Now back propagation is a method that the deep neural networks use to optimize or to train the, uh,
the network.
So the actual thing, what's happening is training is just simply changing the weights and biases.
And in order to match the inputs with the outputs based on the data.
So let's do that.
But let's do it well by hand.
To do so let's consider we have an input value of um of x.
X equals one.
And an output value which is equal y equals or is a bit or.
Y equals two.
Now this is what we have.
And.
We need a network that will have the input multiplied by W and get the output, which is two.
So what do we do is we will have input of x.
This input of X will go to the.
Neural network and.
The neural network or this operation will need w the weight.
So this wait, let's consider its value is or initial value is one.
This is of course the initial value of w x is one, a w or y is two.
And W initially is one.
Now the question is is this one value is correct or not correct?
Well back propagation we need to understand how much we we need to change, how how much we need to
change, or how W is related with our loss or the difference between the input and the output.
So let's go step by step now.
W we need to multiply it by x.
And this is going to be the operation which is multiplication multiplication.
And we make it a bit bigger I think 45.
Or 50.
So here is multiplication and what we will get, we will get.
Y or we call it y hat.
Y hat.
Now y hat will go into another operation Y because we don't know.
We don't know is this y hat is correct or not correct?
We need to compare it with the real value, which is the y value.
So we have the y value which equals of course two.
Or.
Yeah, we can just put it or let's just keep it like this y and it will enter this operation.
And what we need is we need a subtraction.
Subtract y from y hat and we will get.
A value.
Another value is the difference, which is s.
So here is going to be S.
Now after that what do we need.
Usually the difference is might be minus, might be positive.
So what we care about is not really the if it's positive or negative.
So usually we want to square the value.
We want to have a square value of uh well of s.
So we um, this is like for example, the mean square error is actually squaring the error so that we
will do another operation, which is.
That squaring because what we care about or the loss itself, the value itself is, is just if the number
is higher or lower, it's both bad.
So what we care about is to minimize the actual loss, which is the absolute value, or well a squared
value of the error.
So we have to.
Have is to the power of two.
Now after that.
After it got squared.
This is actually.
The value which we call loss value.
Los.
So.
This is the system, like our system.
But until now, we didn't understand what did it.
What did we do?
Like what did we do?
Back propagation.
We didn't till now.
What we will do now is in order to do back propagation, first we need to do what we call a forward
pass and then we go backward pass.
So for this simple system, let's take just a bit um like thick line and let's choose green.
And then w we said it's one x.
This is of course initial value.
And x also equals to one one multiplied one y hat will be equal.
Well y hat it will be.
Or actually this way hat equals one multiplied one which equals to one.
Why?
On the other hand, its value is two, so two multiplied one 2 or 1 minus two will equate to minus one
as x will as s will be minus one.
You square this thing and what you will get is the Los equals one.
So this is quite, you know, a forward pass.
Now we didn't solve the problem yet.
What we need is we need, uh, how much I need to change w to make.
The loss is zero.
Usually it's a minimization problem.
So what should I do?
What you should do is basically apply the chain rule.
And or we need to apply the back propagation.
But actually the back propagation is based on the chain rule.
In order to calculate dz and connected it with w what you need is or the loss with w we calculate.
Let's say we want d over d w d loss over d w.
What it will equal will equal de Los de Los.
Over the years.
This one.
These.
Multiplied this one.
We put a DZ here.
And then divide by y hat.
Dee.
Why not y this one y hat or the predicted y.
Multiplied the hat, the y hat, the y hat over D.
That will you?
And this will equal the de loss over double.
So what we actually did is basically we.
We just if we dz, we kind of expanded the whole thing.
We care about the loss to the W, but we cannot go directly from W to D and from D loss to W.
We have to go through a process, uh, or the process is um, or we know the D loss to the S, we can
connect dz to d y hat and y hat to d w, but we cannot connect directly here.
So we use the chain rule.
And the chain rule is basically if we got the loss over dz and dz here, this and this will cancel each
other and this and this will cancel each other.
And we go to the loss of a D.
So the the idea of chain rule is quite interesting.
And it this is how it works now.
What is the problem?
Now we need to calculate the every, uh, every point here.
What is the derivation of these points.
So basically loss we said is equal s squared.
So if we want to say loss.
Equals s squared.
So what is the derivation of this loss?
Or de?
Well.
Los.
Loss over over d s will be.
Will be what?
Two.
Yes.
So the derivation of de loss over.
A the s is to s.
That's quite useful now.
What if we.
What if we put.
Let's say I will take this orange.
So we already know this is the derivation.
So how much is a s is minus one.
And if we put s in here what is the value we get.
We get equals minus.
Or you will get minus two because or just a little bit.
We just move it a bit after.
Um, well, basically if we apply, uh, this one, it will be, well, okay.
Like I'll just write it minus two.
This is d loss over d s.
This is not d loss over d w because we applied here's we already know the value and we applied in this
equation.
And then we will get minus two.
This is again the loss over d s now.
Let's go back here.
What's equals to d s d?
S over d well y hat.
Y hat.
Of course, it's all partial derivative.
And we put it here a little bit.
Yeah.
The hat what it will equal.
It will equal two.
D hat.
D all.
Well, just make it a bit smaller.
It's too small.
So it will equal d y hat.
Y hat.
Minus y, minus y.
Two D.
Uh, well, y hat, this one, y hat, this one the predicted.
Y hat.
And then what this derivation will derivation will be.
If you'd hat to d hat, it will equal to one and d y minus d y over d y hat is equal to zero.
Because y doesn't change with y hat, y is a constant, so it become zero.
And now we have the derivation of um well the the this derivation.
Now.
How much is d s over d hat?
Well, it is one, because even y hat is one or whatever it is, it's.
This derivation is always awkward equals to one.
So let's write it down.
Is equals to.
One.
What else do we have?
Or yet still we have d y hat over d w.
A what does it equal?
So y hat.
Over the W.
We just write it a little bit?
Maybe.
How about here?
We put it here.
This one is here.
The.
Y hat.
Over d w.
What it will equal.
It will equal.
We cannot derivative because we don't know how much y hat unless we change y hat to an equation.
So it becomes d y hat is w multiplied x, so w multiplied x.
And derive d w.
This is now easier.
Derive this thing to, like, um, again, the y hat over the w.
This one.
When we derive this dw over dw d hat, deriving w to w will equal to one.
That means the final answer will be x.
And that's it.
That's the value we have just a little bit.
I move these things to look a little bit better.
And put this one in its location.
Maybe.
Right.
Here.
So the value of this thing what it will equal this one and put orange x is one we we put the x value
in here.
What do we get.
We get one.
So.
Let's calculate this guy.
Let's use, uh, let's say blue.
De Los over dos.
How much is it?
Minus two.
Minus two multiplied.
DZ over d what its account to dz over d.
A white hat.
Sorry.
It will be one.
Multiply d over DW what it will be d y hat of course over DW it will be one.
What is all of this will be equal to one multiplied one is one multiplied minus two is minus two.
So basically this is our, uh, basically the, uh, the value we have, which is minus two.
The loss will change with w minus with a value of minus two.
That means if the w increased, the loss will decrease.
So what do we need to do?
We need to increase w.
A matter of fact it is correct.
If we put w is two two multiplied one y hat will be two and two multiplied two is going to be equals
to zero, which makes the loss is also equals to zero.
So with this case we understand how can we change w to match the loss.
So this is basically the idea of back propagation.
And this is what in principle, every neural network, um, get trained on.
We just calculate how much every weight is connected to the loss.
And next class we will apply this with using PyTorch.