-: Hello and welcome back to the course on deep learning.
All right, today we're talking
about the activation function.
Let's get straight into it.
So this is where we left off previously.
We talked about the structure of one neuron.
So there it is in the middle.
We know that it has some inputs values coming in,
it's got some weights.
Then it adds up,
it calculates the weight of some of those inputs
and then applies the activation function.
And on step three, it passes on the signal
to the next neuron.
And that's what we're talking about today.
We're talking about the value
that is going to be passed over,
so we're talking about the activation function
that's being applied.
So what options do we have for the activation function?
Well, we're going to look
at four different types of activation functions
that you could choose from.
Of course there are more
different types of activation functions
but these are the predominant ones
that you'll be hearing about
and that we'll be using in this course.
So here is the threshold function.
This is what it looks like.
So on the X axis you have the weighted sum of inputs.
On the Y axis, you have just the values from zero to one.
And basically the threshold function
is a very simple type of function
where if the value is less than zero,
then the threshold function passes on zero.
If the value is more than zero or equal to zero
then threshold function passes on a one.
So it's basically kind of like yes, no type of function.
Very, very straightforward,
very kind of like rigid type of function.
Either yes or no,
no other options.
So there you go, that's how it works.
Very simple function.
Let's move on to something a bit more complex now.
The sigmoid function,
very interesting formula that we have here.
You'll see just now there it is,
one divided by one plus e the power of minus x.
Whereas in this case, of course,
X is the value of the weighted sums.
And so yeah, so that this is what the sigmoid looks like.
It's a function which is used
in the logistic regression if you recall,
from the machine learning course.
So what is good about this function
is that it is smooth.
Unlike the threshold function,
this one doesn't have those kinks in its curve,
and therefore is just nice and smooth, gradual progression.
So anything below zero, it just like drops off,
above zero it approximates towards one.
And this sigmoid function is very useful
in the final layer in the output layer,
especially when you are trying to predict probabilities.
And we'll see that throughout this course.
And then we've got the rectifier function.
Rectifier function, even though it has a kink
is one of the most popular functions
for artificial neural network.
So it goes all the way to zero.
It's zero and then from there it gradually progresses
as the input value increases as well.
And we'll see that throughout the course.
We'll see that in other intuition tutorials
and we'll also see that how we use this function
in the practical side of the course.
And I will comment on this a bit more
in a few slides from now.
So just remember that rectifier function
is one of the most used functions
in artificial neural networks.
And finally, we've got one more function
that you will probably hear about.
It's the hyperbolic tangent function.
It's very similar to the sigmoid function,
but here the hyperbolic tangent function goes below zero.
So the values go from zero to one
or approximately to one,
and go from zero to minus one on the other side.
And that can be useful in some applications.
So we're not going to go into too much depth
on each one of these functions.
I just wanted to acquaint you of them
so that you know what they look like
and what they're called.
If you'd like to get some additional reading,
then check out this paper by Xavier Glorot called,
"Deep Sparse Rectifier Neural Networks" 2011 paper.
And there you will find out
exactly why the rectifier function
is such a valuable function,
why it's so popular used.
But nevertheless,
for now you don't really need to know all of those things.
For now, we just get to start applying them.
We just start using them more and more and more.
And so when you feel comfortable
with the practical side of things,
then you can go and refer to this paper,
and then you'll be able
to soak in that knowledge much quicker
and it'll make much more sense.
So, but just keep this in mind
that when you're ready,
when you feel that you're ready
then you can go and refer to this paper
and get some valuable knowledge from there.
So just to quickly recap,
we have the threshold activation function,
which looks like this.
The sigmoid activation function,
which looks like this.
We have the rectifier function
and we have the hyperbolic tangent function.
And now to finish off this tutorial,
let's quickly do a few exercises.
So we'll just do two quick exercises
to help that knowledge sink in.
So first one is we've got an example here
of a neuro network with just one neuron,
and then right away the output layer.
And the question is,
assuming that your dependent variable is binary,
so it's either zero one,
which threshold function would you use?
So out of the ones that we've discussed;
we have the threshold function, the sigmoid function,
the rectifier function,
and we've got the hyperbolic tangent function.
In their raw forms,
which ones would you be able to use for a binary variable?
Okay, so the answers here are,
there's two options that we can approach this with.
So number one is the threshold activation function,
because we know that it's between zero and one,
and it gives you a zero under certain values,
and then otherwise it gives you one.
So it only can give you two values.
It fits perfectly, fits this requirement perfectly
and therefore you can just say
Y equals the threshold function of your weighted sum.
And that's it.
And then the second case what you could use
is the sigmoid activation function.
It is actually also between zero and one,
just what what we need.
But at the same time, you want just zero one, right?
It's not exactly the what we need,
but in this case what you could use it as
is the probability of Y being yes or no.
So we want Y to be zero one,
but instead we'll say that the sigmoid activation function
tells us the probability of Y being equal to one.
So basically the closer you get to the top,
the more likely it is that this is indeed
a one or a yes, rather than a no.
And yeah, so that's very similar
to the logistic regression approach.
And those are just two examples
of if you have a binary variable.
And now let's have a look at another practical application.
Let's have a look at how all this would play out
if we had a neural network like this.
So in the first input layer we have some inputs.
They're sent off to our first hidden layer,
and then an activation function is applied.
And usually what you would apply here
and what you'll see throughout this course
is we would apply a rectifier activation function.
So it would look something like that.
We apply the rectifier activation function
and then from there,
the signals would be passed on to the output layer
where the sigmoid activation function would be applied.
And that would be our final output.
And that could predict a probability, for instance.
So this combination is gonna be quite common
where in the hidden layers we apply the rectifier function
and then in the output layer we apply the sigmoid function.
So there we go,
hope you enjoy today's tutorial.
Now you are quite well versed
in the four different types of activation functions,
and you will get some hands on practical experience
with them throughout this course.
We'll be using them all over the place,
so you'll get to know them quite intimately
and you should be quite comfortable with them.
But for now, this is the knowledge
that you need to progress and understand
what is going to be happening further down in this course.
And on that note, I look forward seeing you next time.
Until then, enjoy deep learning.