Hi, and welcome back to the lesson on L1 L2 regularization.
Let's take a look at what these methods do, and we'll explore and a contrast to them a bit.
So firstly, they're a type of wit constraining regularization methods.
They both work by forcing all parameters.
That is the weird symbiosis or model or feature maps and max pulleys and fully connected layers.
What does weird symbiosis?
It forces them to take smaller values.
And the reason this reduces overfitting is that by reducing the weights, no network reduces the effects
of those weights on activation function.
So you don't have certain filters being overpowering other filters and when they get activated.
So L1 or L1 norm, also called lasso regression, is basically added to the lost function.
Yeah.
So you can see there are a few parameters that go into this formula here.
Let's take a look at what they are.
So Biji, which is where we take in a modular, so absolute value of Biji, all the weights and lambda
is a concern that controls the effect of the penalty applied, usually less than one.
So a large lambda means that we're making a penalty term quite large.
You consider this because it scales proportional to this, and L1 uses an absolute value as a penalty.
So what is does here, basically by putting the weights and putting this this formula in the weeds here
in the lost function, we're effectively adding that to a lost function and using that as a metric to
update or weights in the network.
Now let's take a look at L2 regularization, and you can immediately see it's no longer the absolute
value.
Like the L1, it's no the square value and biji again.
Oh, and biases lambda controls the effect of the penalty again.
And similarly, a large lambda controls how much or how big if we want our penalty to be.
And importantly, here, L2 uses a squared magnitude up to be it as a penalty.
This can immediately tell you that if it's if it's hitting a square value, it's going to put a big
emphasis on bigger weights and making them smaller in the end.
So the differences between them for the formulas here.
So you can contrast them is that L1 shrinks the weights by a constant amount towards zero by shrinking
less important weights, which are weights to zero.
This sort of acts like a feature selection algorithm, which is quite good, quite desirable, yielding
sparse models of in L2, which shrink by a proportional to the weights and L2.
As I said, because of the square penalises large weights, more stunned smaller weights and smaller
weights.
Less so L1 tilt tends to concentrate the weight of the network to a relatively small number, but highly
important connections.
So you can see that the boot would work in different ways.
So let's take a summary of L1, L2, so L1 and L2 regularization, both macro and that would prefer
to live in the smaller weights tends to push the weights towards smaller values.
Large weights are actually loud sometimes, but only if the considerably improve the first part of our
cost function or lost function.
This can be interpreted as a way of compromising between finding small weights and minimizing the original
lost function.
So it's basically a added term to a lost function that we have to consider now.
And the degree of compromise depends on lambda.
A small lambda means that we prefer to minimize the original lost function.
A larger lambda means we prefer we're putting emphasis.
We go back to the formula.
A large lambda means that we're putting more emphasis on this.
Where it's a small lambda means you're putting more emphasis on the lost function itself.
That's it for this chapter.
Next, we'll take a look at the drop permitted, which is another very useful regularisation metaphor,
so I'll see you in the next section.
Thank you.