In this lecture we are going to discuss activation functions.
Previously we learned that the sigmoid function allows us to build neural networks and it does several
important things for us.
First it maps its inputs to go from zero to 1.
This is nice because it mimics the biological neuron.
So our neural network is literally a network of artificial neurons.
Second and this is more practical.
It makes our neural network nonlinear.
The function itself is non-linear and it also makes it so that we can't reduce the neural network into
a simple linear equation.
At the same time in modern Deep Learning we've discovered that there are some problems with the sigmoid.
And so it is no longer used as often except in some specific cases
if you recall something we discussed earlier was the importance of standardization.
We don't want to have one input in the range one million to 5 million and then another input going from
zero to zero point 0 0 0 1 instead we would like to have all of our data centered around zero and in
approximately the same range.
Now the sigmoid is problematic in this regard.
If you recall the output of a sigmoid is always between 0 and 1 and its middle value is therefore zero
point five.
Therefore the output of a sigmoid can never be centered around zero.
This goes back to our idea of the uniformity of a neuron that work one layer of a neuron that where
it takes as input the output of the previous layer.
So if the previous layer is outputting numbers centered around zero point five that's not quite right.
This is because this output is going to become the input to the next layer and the next layer wants
to also see inputs which are standardized.
Therefore we want both the inputs and the output of each layer to be standardized
luckily there is a solution to this although it takes away from this idea that a neuron network is made
up of this idealized neuron simulation instead of using the sigmoid we'll use a function that has the
exact same shape as the sigmoid just centered around zero.
This function is called the 10 H which is short for hyperbolic tangent if you recall there are hyperbolic
versions of all the trigonometric functions.
So there is a hyperbolic sine hyperbolic cosine and so forth.
Well it turns out that the hyperbolic tangent analogous lead to the trigonometric tangent is just the
hyperbolic sine over the hyperbolic cosine and it has the equation you see here the sigmoid has a similar
equation one divided by one plus the exponent of minus X.
The major difference between the sigmoid and the 10 H is that the sigmoid goes between 0 and 1 while
the 10 age goes between minus 1 and plus 1 as an exercise.
You may want to try and prove the relationship between the sigmoid and the 10 H in particular that the
10 H is just a scaled and vertically shifted version of the sigmoid
now although the teenagers is a little better than the sigmoid the story is not over in modern deep
learning.
Researchers have figured out that there is actually a problem with both of these activation functions.
I like to tell this story as a bit of a cautionary tale for a long time.
Researchers were very attached to the beauty of neural networks as is they liked this idea of a uniform
neural network made up of idealized neurons.
They like this idea that the sigmoid and hence also the 10 H with smooth differential functions.
It is mathematically convenient and beautiful.
The problem is they don't work that well but nobody wanted to break away from this classic way of doing
things
the major problem with sigmoid and tan ages is called The Vanishing gradient problem.
Now this part requires a little bit of calculus knowledge.
So if you're not comfortable with that then feel free to skip ahead.
Remember that our method of training a neural network is gradient descent.
And obviously this involves finding the gradient of the cost with respect to the parameters.
The problem is when you have a very deep neural network your gradient has to propagate backwards throughout
the neuron that we're starting from the end.
What happens is your output is made up of a bunch of composite functions basically a sigmoid of a sigmoid
of a sigmoid and so on when you take the gradient of composite functions you get the chain rule so composite
functions become multiplication in the derivative.
So what does this end up looking like.
Well the further you go back in the neural network the more terms you have in the chain rule to multiply.
So you're multiplying by the derivative of the sigmoid.
Over and over again what happens when you do that.
Why is the derivative of the sigmoid a problem
well consider what the sigmoid actually looks like most of the sigmoid is flat.
In other words the derivative of the sigmoid is very nearly zero at most points only in the center is
it non-zero.
It also turns out that the maximum value of the derivative is only zero point two five.
This is the highest possible value of the derivative of the sigmoid.
Okay so why is that a problem.
Well what happens when you multiply numbers that are very small.
The answer is you get an even smaller number.
Let's say you multiply zero point two five.
The maximum possible value five times the result is zero point to five to the power five which is approximately
zero point zero zero one.
What happens if more realistically you have a value like zero point one multiplied by itself five times.
That's zero point once the Power Five which is zero point zero zero zero zero one.
In other words this leads the result that the further you go back in the neural network the smaller
the gradient becomes.
We call this the vanishing gradient problem.
So how does this problem manifest.
We'll take a look at this graph which shows the magnitude of the gradient at each layer of the neuron
that work as it is trained.
You'll notice that the further you go back the smaller the gradient gets.
Remember that the training algorithm is to take small steps in the direction of the gradient.
Well if the gradient is nearly zero that means the update to the weights is also nearly zero.
The end result is that weights close to the input of the neuron that work are almost not trained at
all.
This was a problem in the olden days which prevented us from building very deep neural networks like
we have today which can have hundreds of layers
back in the day.
There was lots of theory around how to solve this problem.
One method was called a greedy layer wise pre training which was invented by Geoffrey Hinton and his
students.
Since all the layers of the neural network could not be trained all at once the idea was you could train
each layer one at a time so what you would do is train the first layer using some alternative lost function.
By the way if you want to know what that last function is it's basically an auto encoder but hold that
thought for now as it's not an important detail then once you were done training the first layer you
would add on a second layer and train that layer by itself not touching the first layer anymore.
Then you would add a third layer and train only the third layer leaving the first two layers alone.
Once you got to the last layer all of the previous layers would already be trained to some extent.
Now despite all this theory which generated lots of theoretical research into new models such as restricted
boltzmann machines deep Boltzmann machines and so forth today we no longer require such complicated
models in order to train very deep and learn that works although that's not to say they're not worth
learning about.
In fact the solution was simple Just don't use activation functions that have vanished ingredients.
So throw away these beautiful smooth differential functions like the Tanakh and the sigmoid instead.
Let's use this ugly looking not completely differential function called the real you the real you is
short for rectify a linear unit.
Basically it looks like a hockey stick and it has a corner at zero where the derivative is technically
not defined.
The fact that value is greater than zero never have a zero gradient makes training neuron that works
a lot more efficient.
Now you might be wondering wait a minute.
If a zero gradients are so bad then why does the real you work at all.
It appears that half the function any input less than zero has a derivative that is exactly zero.
Never mind vanishing the gradient is already vanished.
Indeed this is somewhat of a problem as it leads to a phenomenon called Dead neurons.
The neurons are neurons that always output zero because the weighted sum of its inputs are always less
than or equal to zero.
However and this is important in deep learning what we care about is experimental results.
In other words the most important thing is that it works not how theoretically satisfying is because
as we now know that can lead to lots of wasted time.
The fact that just the right side doesn't vanish seems to be good enough according to what we've observed.
Of course some researchers have tried to make modifications to the simple rule to solve this problem
of dead neurons.
One alternative is the leaky rescue which has a slope of less than one like zero point one for values
less than zero.
Importantly this is still a nonlinear function so we're able to learn non-linear geometrical patterns
using the leaky well you your derivatives will always be positive.
Just like the sigmoid and the tan age
there are other options as well such as the ACLU or exponential linear unit which has a more steadily
decreasing value on the left side.
The authors claim that this activation function speeds up learning and leads to higher accuracy than
the real you.
One interesting aspect of the ACLU is that it allows its outputs to be negative which goes back to the
idea that we like the mean of the values to be close to zero
another option which is very similar is the soft plus activation which because you're taking the log
of the exponent looks very linear when the input is reasonably large now for both of these previous
activation functions.
There is the vanishing gradient on the left side but we've established that it's not so much of a problem
since we know that the real you already works and it has gradients that are equal to zero also.
Although we initially stated that we would like the inputs at each layer to be centered around zero
we can see that the real U.N. soft laws do not accomplish this for the soft plus and the real you the
minimum value is zero while the maximum value is infinity.
This definitely means they won't be centered around zero.
So is the real you not a good choice in the end.
Now despite all this work to find alternatives to the value activation these days most people still
use the value as a reasonable default choice.
It works well and sometimes you'll find that using other alternatives such as the leaky you or the evil
you offer no benefit.
Sometimes they do.
Which is why you always have to experiment for yourself.
My motto which a lot of my students are tired of hearing by this point is that machine learning is experimentation
and not philosophy.
Never use your mind to try and predict the outcome of a computer program.
If you have a computer that is always the suboptimal course of action why not simply run the computer
program with a computer.
Your mind is not suitable for running computer programs but computers are therefore follow the rule.
Don't use philosophy use experimentation
interestingly some researchers have talked about the biological plausibility of the real you activation
function.
In fact it may even be more biologically plausible than the sigmoid to understand this you have to understand
a little more about how action potentials encode information.
What is the difference between an action potential.
When you hear a quiet sound versus an action potential.
When you hear a loud sound.
The answer is there is no difference.
An action potential is just an action potential.
The key really is in the frequency of action potentials.
When you hear a very quiet sound your neurons are only activated a little bit so you'll get some action
potentials but they won't be very frequent.
If you hear a very loud sound like say you're at a concert or a party then your neurons are getting
lots of stimulation.
The action potentials are going to be very frequent.
In other words closer together in time.
So what we're saying is the more intense a stimulus is the higher the frequency of the action potentials.
We call this frequency coding
what does this mean in terms of the real you.
Well you can think of it as the real you is just encoding the actual potential frequency itself.
Of course the minimum frequency is zero and it's not possible to have a negative frequency.
That's why the minimum value of the value is zero.
Then as the inputs into the neuron get larger because they are being stimulated more the receiving neuron
also gets stimulated more increasing its frequency
now.
One thing to note is that we can go even deeper than this although this is not part of mainstream deep
learning quite yet.
As a side note if you don't want to listen to this part it's not required.
It's something that I think is interesting to those of us who are interested in modelling brains.
But if you are a hardcore statistician then perhaps you may think differently what we know based on
neuroscience is that unlike the real you activation action potential frequency is not linearly related
to its input stimuli.
Instead we have a linear relationship.
It can be modeled using the log function or a root function like the square root a good way to understand
this is with sound something that is twice as loud does not sound twice as loud.
Using measurements of sound intensity this is the intuition behind the decibel scale.
As you recall the decibel scale is a logarithmic scale.
For example a 90 decibels sound would be like sitting inside a moving bus 100 decibels sound would be
like standing by an electric saw at 110 decibels sound would be like a loud orchestra of 120 decibels
sound would be like standing near a jet engine.
A 130 decibels sound would be like being close to artillery fire and this value is the threshold of
pain.
So when you hear a sound this loud it actually hurts physically.
To put that in perspective 100 decibels is ten times more powerful than 90 decibels 110 decibels is
100 times more powerful 120 decibels is 1000 times more powerful and 130 decibels is ten thousand times
more powerful
some work has been done to experiment with activation functions that more accurately model the frequency
characteristics of real neurons such as the bee are you.
But as a whole they haven't yet caught on in the Deep Learning Community.
I've attached the paper on the bee are you two extra reading ATX t in case you want to check it out.
The author reports that the BYU activation function led to better results than the real you and the
ACLU.
So it may be something worth trying in your own project.