So now let's take a look at a final regularization technique.
And this one's called batch normalization, which is a very, very good technique.
However, it's a bit tricky to understand, but it's also very simple to implement suits, give and
take.
So firstly, before I explain batch normalization, let's talk about some deep learning problems.
Firstly, training deep learning networks is slow, requires a lot of tweaking parameter adjustments,
little adjustments, a lot of experimentation in general, and obviously lots of data.
So remember, models are updated layer by layer backwards to what the input right to left using back
propagation.
This assumes that a width adjusted into probably a fixed.
So here's a code it took from the Deep Learning Handbook page 317.
I'll read it out for you.
Remember this so very deep models involve the composition of several functions or layers.
The gradient tells how to update each parameter under the assumption that utterly is to not change.
That's what I'm seeing here in practice.
Do we update all the layers simultaneously?
So that's a fallacy in a way.
So take a look at remember back propagation works.
This is just a quick recap to show you that it's moving right to left.
So it's challenging doing treating it as inputs changes the parameters of the previously themselves
change.
This results in slower training as it requires smaller learning rates and careful parameter in initialization.
And we have to go through all of this again, but that's OK.
Just a nice little graphical illustration of back propagation.
So what does batch normalization?
Well, that's known for sure.
It is a technique that helps coordinate the object of multiple layers in a model, so it standardizes
by reskilling it.
So we have a mean of zero and a standard deviation of one.
The activations of two prior LEO, thus scaling the output of the Leo IT repositories as the model to
make some units always be standardized by definition.
This allows it to reduce internal covariance shift.
Now this is getting a bit complicated, so I don't expect you guys to follow all of this.
However, if you want to get a deeper understanding or have a deeper understanding of batch normalization,
I'd encourage you to read this paper here called batch normalization.
You can find it at this link.
This URL here and this is how the what is defined, what internal covariance shift is.
We define internal coverage shift as a change in the distribution of network activations due to the
change in that would parameters during training, but confusing.
I admit, however, just remember that's known as a technique that helps coordinate the update of multiple
layers in a model.
It's by doing that, and it does it by actually finding summarizing the outputs of the activations of
the previous layer.
That's basically what it does.
So let's take a look at how we implement batch normalization.
So recall that output of a convolutional layer is four dimensional.
It has but size, feature, map height, feature map width and the number of channels, meaning the
depth so it calculates and botched on the mean and the standard deviation of each input variable associated
with that layer per mini batch and uses this to perform the centralization.
In CNN's Remember filter, we had shear.
That's the same field as applied to all parts of the image, which is a good thing.
Therefore, this means that the mean and standard deviation are taken for each filter per minute batch,
giving us either tree means and standard deviations, or one if it's a grayscale image.
So here's some advice in how to use batch on properly.
Maximum was actually is super easy to implement.
It's as easy as just adding a bottom layer in cameras and similar impact by to watch as well.
So when we using it would drop out, this is a recommendation or the that we do these things, we have
a layer matching real robots.
Then another country, if you want to continue having more layers.
So generally it's used between the conflict and the activation function, as you can see here.
So it will stop there for now.
And next, we'll take a look at how and when we use a regularization.
So, I mean, I'll tell you generally, you should always use it.
But there are some specific cases where you may want to use it or may not want to use it.
So we'll take a look at that in the next section.
Thank you.