In this lecture, we are going to discuss activation functions.
Previously, we learn that the sigmoid function allows us to build neural networks and it does several
important things for us.
First, it maps its inputs to go from zero to one.
This is nice because it mimics the biological neuron.
So our neural network is literally a network of artificial neurons.
Second, and this is more practical, it makes our neural network nonlinear.
The function itself is non-linear and it also makes it so that we can't reduce the neural network into
a simple linear equation at the same time in modern deep learning.
We've discovered that there are some problems with the sigmoid and so it is no longer used as often,
except in some specific cases.
If you recall, something we discussed earlier was the importance of standardization, we don't want
to have one input in the range, one million to five million and then another input going from zero
to zero point zero zero zero one.
Instead, we would like to have all of our data centered around zero and in approximately the same range.
Now, the sigmoid is problematic in this regard.
If you recall, the output of a sigmoid is always between zero and one and its middle value is therefore
zero point five.
Therefore, the output of a sigmoid can never be centered around zero.
This goes back to our idea of the uniformity of a neural network.
One layer of a neural network takes as input the output of the previous layer.
So if the previous layer is outputting numbers centred around zero point five, that's not quite right.
This is because this output is going to become the input to the next layer and the next layer wants
to also see inputs which are standardized.
Therefore we want both the inputs and the output of each layer to be standardized.
Luckily, there is a solution to this, although it takes away from this idea that a neural network
is made up of this idealized neuron simulation.
Instead of using the sigmoid, we'll use a function that has the exact same shape as the sigmoid just
centered around zero.
This function is called the Tarnak, which is short for hyperbolic tangent.
If you recall, there are hyperbolic versions of all the trigonometric functions.
So there's a hyperbolic sine, hyperbolic, cosine and so forth.
Well, it turns out that the hyperbolic tangent analogously to the trigonometric tangent is just the
hyperbolic sine over the hyperbolic cosine.
And it has the equation you see here.
The sigmoid has a similar equation, one divided by one plus the exponent of minus X, the major difference
between the sigmoid and the H is that the sigmoid goes between zero and one, while the Tanach goes
between minus one and plus one.
As an exercise, you may want to try and prove the relationship between the sigmoid and the Tanach in
particular, that the ten is just a scaled and vertically shifted version of the sigmoid.
Now, although the teenager is a little better than the sigmoid, the story is not over in modern deep
learning.
Researchers have figured out that there's actually a problem with both of these activation functions.
I like to tell this story as a bit of a cautionary tale.
For a long time, researchers were very attached to the beauty of neural networks, as is.
They like this idea of a uniform neural network made up of idealized neurons.
They like this idea that the sigmoid and hence also the Tanach were smooth differentiable functions.
It is mathematically convenient and beautiful.
The problem is they don't work that well, but nobody wanted to break away from this classic way of
doing things.
The major problem with sigmoidal and tanagers is called the vanishing gradient problem.
Now this part requires a little bit of calculus knowledge.
So if you're not comfortable with that, then feel free to skip ahead.
Remember that our method of training a neural network is gradient descent.
And obviously this involves finding the gradient of the cost with respect to the parameters.
The problem is when you have a very deep neural network, your gradient has to propagate backwards throughout
the neural network starting from the end.
What happens is your output is made up of a bunch of composite functions, basically a sigmoid, of
a sigmoid, of a sigmoid and so on.
When you take the gradient of composite functions, you get the chain rule.
So composite functions become multiplication in the derivative.
So what does this end up looking like?
Well, the further you go back in the neural network, the more terms you have in the chain rule to
multiply.
So you're multiplying by the derivative of the sigmoid over and over again.
What happens when you do that?
Why is the derivative of the sigmoid a problem?
Well, consider what the sigmoid actually looks like, most of the sigmoid is flat.
In other words, the derivative of the sigmoid is very nearly zero at most points.
Only in the center is it non-zero.
It also turns out that the maximum value of the derivative is only zero point two five.
This is the highest possible value of the derivative of the sigmoid.
OK, so why is that a problem?
Well, what happens when you multiply numbers that are very small?
The answer is you get an even smaller number.
Let's say you multiply zero point to five, the maximum possible value five times.
The result is zero point twenty five to the power of five, which is approximately zero point zero zero
one.
What happens if more realistically, you have a value like zero point one multiplied by itself?
Five times, that's zero point one, the power of five, which is zero point zero zero zero zero one.
In other words, this leads to the result that the further you go back in the neural network, the smaller
the gradient becomes.
We call this the vanishing gradient problem.
So how does this problem manifest?
Well, take a look at this graph, which shows the magnitude of the gradient at each layer of the neural
network as it is trained.
You'll notice that the further you go back, the smaller the gradient gets.
Remember that the training algorithm is to take small steps in the direction of the gradient.
Well, if the gradient is nearly zero, that means the update to the weights is also nearly zero.
The end result is that weights close to the input of the neural network are almost not trained at all.
This was a problem in the olden days which prevented us from building very deep neural networks like
we have today, which can have hundreds of layers.
Back in the day, there was lots of theory around how to solve this problem.
One method was called greedy layaways pre training, which was invented by Geoffrey Hinton and his students,
since all the layers of the neural network could not be trained all at once.
The idea was you could train each layer one at a time.
So what you would do is train the first layer using some alternative lost function.
By the way, if you want to know what that lost function is, it's basically an auto encoder.
But hold that thought for now as it's not an important detail.
Then once you were done training the first layer, you would add on a second layer and that layer by
itself not touching the first layer anymore.
Then you would add a third layer and train only the third layer, leaving the first two layers alone.
Once you got to the last layer, all of the previous layers would already be trained to some extent.
Now, despite all this theory, which generated lots of theoretical research into new models such as
restricted Boltzmann machines, deportment, machines and so forth, today we no longer require such
complicated models in order to train very deep neural networks, although that's not to say they're
not worth learning about.
In fact, the solution was simple.
Just don't use activation functions that have vanishing gradients.
So throw away these beautiful, smooth differentiable functions like the Tanach and the sigmoid.
Instead, let's use this ugly looking, not completely differentiable function called The Real You.
The Real You is short for rectifier linear unit.
Basically it looks like a hockey stick and it has a corner at zero where the derivative is technically
not defined.
The fact that value is greater than zero never have a zero.
It makes training neural networks a lot more efficient.
Now, you might be wondering, wait a minute, if a zero gradients are so bad, then why does the rescue
work at all?
It appears that half the function, any input, less than zero has a derivative that is exactly zero.
Never mind vanishing.
The gradient is already vanished.
Indeed, this is somewhat of a problem as it leads to a phenomenon called dead neurons.
Dead neurons are neurons that always outputs zero because the weighted sum of its inputs are always
less than or equal to zero.
However, and this is important in deep learning, what we care about is experimental results.
In other words, the most important thing is that it works, not how theoretically satisfying it is,
because as we now know, that can lead to lots of wasted time.
The fact that just the right side doesn't vanish seems to be good enough according to what we've observed.
Of course, some researchers have tried to make modifications to the simple rescue to solve this problem
of dead neurons.
One alternative is the leaky rescue, which has a slope of less than one like zero point one for values
less than zero.
Importantly, this is still a nonlinear function.
So we're able to learn nonlinear geometrical patterns using the leaky well, you your derivatives will
always be positive, just like the sigmoid and the Tanach.
There are other options as well, such as the ACLU or exponential linnear unit, which has a more steadily
decreasing value on the left side.
The authors claim that this activation function speeds up learning and leads to higher accuracy than
the value.
One interesting aspect of the ACLU is that it allows its outputs to be negative, which goes back to
the idea that we like the mean of the values to be close to zero.
Another option which is very similar is the soft plus activation, which, because you're taking the
log of the exponent, looks very linear when the input is reasonably large.
Now, for both of these previous activation functions, there is the vanishing gradient on the left
side, but we've established that it's not so much of a problem since we know that the value already
works and it has gradients that are equal to zero.
Also, although we initially stated that we would like the inputs at each layer to be centered around
zero, we can see that the U.N. supplies do not accomplish this for the supplies and the you the minimum
value is zero, while the maximum value is infinity.
This definitely means they won't be centered around zero.
So is the value not a good choice in the end?
Now, despite all this work to find alternatives to the rescue activation these days, most people still
use the rescue as a reasonable default choice.
It works well, and sometimes you'll find that using other alternatives, such as the leaky Riu or the
ACLU offer no benefit.
Sometimes they do, which is why you always have to experiment for yourself.
My motto, which a lot of my students are tired of hearing by this point, is that machine learning
is experimentation and not philosophy.
Never use your mind to try and predict the outcome of a computer program.
If you have a computer that is always the suboptimal course of action, why not simply run the computer
program with a computer?
Your mind is not suitable for running computer programs, but computers are therefore follow the rule.
Don't use philosophy, use experimentation.
Interestingly, some researchers have talked about the biological plausibility of the rescue activation
function.
In fact, it may even be more biologically plausible than the sigmoid.
To understand this, you have to understand a little more about how action potentials encode information.
What is the difference between an actual potential when you hear a quiet sound versus an action potential,
when you hear a loud sound?
The answer is there is no difference and action potential is just an action potential.
The key really is in the frequency of action potentials.
When you hear a very quiet sound, your neurons are only activated a little bit.
So you'll get some action potentials, but they won't be very frequent.
If you hear a very loud sound like, say you're at a concert or a party, then your neurons are getting
lots of stimulation.
The action potentials are going to be very frequent, in other words, closer together in time.
So what we're saying is the more intense a stimulus is, the higher the frequency of the action potentials.
We call this frequency coding.
What does this mean in terms of the rescue?
Well, you can think of it as the rescue is just encoding the actual potential frequency itself.
Of course, the minimum frequency is zero and it's not possible to have a negative frequency.
That's why the minimum value of the rescue is zero.
Then, as the inputs into the neuron get larger because they are being stimulated more, the receiving
neuron also gets stimulated more, increasing its frequency.
Now, one thing to note is that we can go even deeper than this, although this is not part of mainstream,
deep learning quite yet.
As a side note, if you don't want to listen to this part.
It's not required.
It's something that I think is interesting to those of us who are interested in modeling brains.
But if you are a hardcore statistician, then perhaps you may think differently.
What we know based on neuroscience is that unlike the real you activation, actual potential frequency
is not linearly related to its input stimuli.
Instead, we have a nonlinear relationship.
It can be modeled using the log function or a root function like the square root.
A good way to understand this is with sound, something that is twice as loud, does not sound twice
as loud using measurements of sound intensity.
This is the intuition behind the decibel scale.
As you recall, the decibel scale is a logarithmic scale.
For example, a 90 decibels sound would be like sitting inside a moving bus.
100 decibels sound would be like standing by an electric saw.
A 110 decibels sound would be like a loud orchestra at 120 decibels.
Sound would be like standing near a jet engine.
A 130 decibels sound would be like being close to artillery fire, and this value is the threshold of
pain.
So when you hear a sound this loud, it actually hurts physically.
To put that in perspective, 100 decibels is 10 times more powerful than 90 decibels.
110 decibels is 100 times more powerful, 120 decibels is 1000 times more powerful and 130 decibels
is 10000 times more powerful.
Some work has been done to experiment with activation functions that more accurately model the frequency
characteristics of real neurons such as the Bee are you?
But as a whole, they haven't yet caught on in the deep learning community.
I've attached the paper on the bee.
Are you to extra reading that text in case you want to check it out.
The author reports that the Bahru activation function led to better results than the you and the ACLU.
So it may be something worth trying in your own project.