Now, let's take a look at another regularization technique called dropout.
So what is dropout?
Well, it's a different regularization technique and unlike L1 L2 normalization, dropout doesn't modify
the loss function.
Instead, it modifies the network itself.
It works by randomly dropping out sets of lawyers by setting to zero number of nodes in the feature
maps of that layer during training.
So let's take a look at how it is implemented, how this algorithm is implemented.
So the dropout process here is illustration of what's happening here.
This is a neural network here.
We have all in the woods.
This is just a fully connected nodes here.
This example.
And what Drapeau does during training time, we start by randomly choosing half over nodes or whatever
value we set.
This is a parameter that half is what we talk about for now in our network to delete or turn off temporarily.
So what that means is that when we found propagate our mini-Budget through the model and back propagate
together, gradients were only doing it for a modified half of the network right now after for the next
menu batch.
We then restore order nodes and randomly delete another set of nodes for the next mini batch.
So as you can see in every training cycle, a mini batch, different nodes are being trained and different
notes are being turned off.
Well, so what effect does this have?
Well, firstly, the fraction of the amount of nodes a drop of drop out depends on the drop out rate,
which is basically a parameter we set the dropout to.
Usually, it's zero point three, which means that every training mini batch we're dropping 0.3 or three
percent of our nodes here.
But how does dropout achieve this?
Well, achieve this generalization and reduction of overfitting?
Well, it forces the network to lean more robust and reliable features because it acts like we trained
several different networks during the training process.
So intuitively, it's a very, very good way of getting a model to generalize quite well and without
putting too much emphasis on certain features, which would cause a model to have a fit.
But it effectively double doubled the number of iterations required to make or model converge.
When we say converge, we mean we basically mean our model to find a global minimum or at least get
very close to it.
So the longer it takes, the more training time we need, a more GPU power and an actual time we need.
So it's not always a good thing, but generally I would always use drop out when treating a model in
testing to just remember this, we use all activation because all the nodes.
But reduce them by a factor of PE to account for the missing activations during training.
That's important to note.
It's not really that important to be aware of because behind the scenes to pay too much and the celebrities
are taking care of all of those things for us.
But nevertheless, this is an important thing to know.
And just in case your mind is curious about it, we care.
Next, we'll take a look at a very, very useful regularization technique, which is called data augmentation.
So I'll see you in the next lesson.
Thank.