Instructor: Hello and welcome back
to the course on deep learning.
Today we're talking about the neuron,
which is the basic building block
of artificial neural networks.
So let's get started.
Previously we saw an image which looked like this
and these are actual real life neurons
which have been smeared
onto a glass and colored a little bit
and they are observed through a microscope.
So this is what they look like, as you can see
quite an interesting structure, a body,
and then lots of different tails,
kind of branches coming out of them.
And this is very interesting, but the question is
how can we recreate that in a machine
because we really need to recreate it in a machine,
since the whole purpose of deep learning
is to mimic how the human brain works
in the hope that by doing so
we are going to create something amazing.
We're gonna create an amazing infrastructure
for machines to be able to learn.
And why do we hope for that?
Well, because the human brain
just happens to be one of the most powerful learning tools
on the planet or learning mechanisms on the planet.
And we just hope that if we recreate that,
we'll have something as awesome as that.
Our challenge right now,
our very first step to creating artificial neural networks
is to recreate a neuron.
So how do we do that?
Well, first of all
let's have a closer look at what it actually is.
This image was first created
by Spanish neuroscientist Santiago Ramón y Cajal in 1899.
And what he did was he dyed neurons
in actual brain tissue
and looked at them under a microscope.
And while he was looking at them
he actually drew what he saw.
And this is what he saw.
He saw two neurons
or two large neurons over there at the top,
which had all these branches coming out of them
towards their top parts.
And then each one of 'em had this rod
or thread coming out towards the bottom,
very long one.
And yeah, so that's what he saw.
And now, you know, technology has advanced quite a lot
and we have seen neurons much closer, in more detail
and now we can actually draw
what it looks like diagrammatically.
So let's have a look at that.
Here's a neuron.
This is what it looks like
very similar to what Santiago Ramón drew over here.
And here in this neuron,
what we can see is that it's got a body,
that's the main part of the neuron,
and then it's got some branches at the top
which are called dendrites,
and it's also got an axon,
which is that long tail of the neuron.
And so what are these dendrites for
and what's the axon for?
Well, the key point to understand here is
that neurons by themselves are pretty much useless.
It's like ant, right?
An ant on its own, can't do much.
Five ants together, maybe they can pick something up
but again, they can't build an ant hill,
they can't establish a colony.
They can't work together as a huge organism.
But at the same time, when you have lots and lots of ants,
like you have a million ants, they can build a whole colony.
They can build an ant hill.
Same thing with neurons.
By itself, it's not that strong
but when you have lots of neurons together,
they work together to do magic.
And how do they work together?
That's the question.
Well, that's what the dendrites and the axon are for.
So the dendrites are kind of like the receivers
of the signal for the neuron.
And the axon is the transmitter
of the signal for the neuron.
And here's an image of how it all works conceptually.
So at the top, you've got a neuron
and you can see that its dendrites are connected to axons
of other neurons that are even further away above it.
And then the signal from this neuron travels down its axon
and connects or passes onto the dendrites
of the next neuron.
And that's how they're connected.
And in that small image over there,
you can see that the axon doesn't actually touch
the dendrite.
A lot of machine learning
or like a few machine learning scientists
are very adamant about the fact that it doesn't touch.
It doesn't touch, it has been proven
that there is no physical connection there.
But the point that we are interested in is that
that connection between them,
that the whole concept of the signal being passed,
that's called the synapse.
You can see over there in that little image
that figure bracket is a synapse.
And that's the term we're going to be using.
So instead of calling our artificial neurons
the lines that we're gonna have
or the connectors for artificial neurons
we're not gonna be calling 'em axons or dendrites
because then the question is, whose connection is this?
Is it that neuron's or is it this neuron's?
We just call them synapses
and that's kind of just answers all questions right away.
That's just basically where the signal is passed,
doesn't matter who that element belongs.
So that's just a representation of the signal being passed.
And we'll see that just now.
So basically that's how a neuron works.
And yeah, so let's move on to how we going to
represent neurons or how create neurons in machines.
So now we're moving away from neuroscience
and moving into technology.
And here we go.
So here's our neuron also sometimes called the node.
The neuron gets some input signals
and it has an output signal.
So dendrites and axons, remember,
but again, we're gonna call these synapses.
Then these input signals, we're gonna represent them
with other neurons as well.
So in this specific case, you can see that this neuron,
this green neuron is getting signals from yellow neurons.
And in this course we're going to try and stick to
a certain color coding regime where yellow means
an input layer.
So basically all of the neurons that are on the outer layer
or in the first front of where the signal's coming in.
And by signal it might be
a bit of an over overkill to call this a signal.
It's just basically input values.
So you know how even in a simple linear regression,
you have input values
and then you have a, a predicted value.
Same thing here.
So you have input values
and there they are the yellow ones.
And then on the right you'll see just now it'll be red,
it'll be the output value.
The thing that I wanted to point out here is
that in this specific example, we're looking at a neuron,
which is getting its signals from the input layer neurons.
So they're also neurons
but they're the input layer neurons.
Sometimes you'll have neurons which get their signal
from other hidden layer neurons
so from other green neurons.
And the concept is gonna be exactly the same.
It's just in this case, for simplicity sake,
we're portraying this example.
And in terms of the input layer,
the way to think about it is,
in the analogy of the human brain,
the input layer is your senses, right?
So whatever you can see, hear, feel, touch or smell.
And of course, there's a lot of things you can see.
There's a lot of information coming in,
but that's what your brain is limited to.
It's pretty much lives in a box made out of bones.
And it's a mind blowing concept
to think about that your brain
is just locked in a black box
and the only thing-
You can't see, you can't hear,
the only thing it's getting is electrical impulses
coming from these organs that you have,
which are called your ears, nose, eyes, you know,
your sense of touch and whatever
and your taste, right?
So it's just getting signals,
but it basically lives in this dark black box
and it's making sense of the world through your senses.
It's phenomenal.
And so, yeah, so you have these inputs that are coming in,
in the terms of human brain, those are your five senses.
And in terms of machine learning or deep learning
that is basically your input values.
So your independent variables
and we'll get that too in a second.
So your input values,
the signal is passed through synapses to the neuron
and then your neuron has an output value
that it passes further on down the chain.
In this specific case,
in terms of color-coding, again, yellow means input layer.
So we're kind of simplifying everything here.
We're saying we're only gonna have like the input layer
then we're gonna have one hidden layer with the green
which is the hidden layer.
And then we're gonna have right the output layer right away.
So just so that we can get used to these colors for now.
So there we go.
That's the basic structure.
So now let's look in a bit more detail
at these different elements that we have.
So we've got the input layer, and what do we have here?
Well, we have these inputs
which are in fact independent variables.
So independent variable one, independent variable two
independent variable m.
The important thing to remember here is that
these independent variables are all
for one single observation.
So think of it as just one row in your database.
One observation.
You just take all of the independent variables,
you know, maybe it's the age of the person?
The amount of money in the bank accounts?
And then how do they drive or walk to work?
What method of transportation do they use?
So, but that's all descriptors of one specific person
that you are either you're training your model on
or you're performing some prediction on.
And the other thing you need to know
about these variables is that you need to standardize them.
So you need to either standardize them
which means make sure that they have a mean
of zero and variance one,
or you can also sometimes and Hadelin will point out
these traces in a bit more detail.
Perhaps in the practical tutorials,
you might come across these,
sometimes you might want to, not standardize,
you might wanna normalize them,
meaning that instead of making sure the mean is zero
and variance is one, you just subtract the minimum value
and then you divide by the maximum minus minimum,
so by the range of your values,
and therefore, you get values between zero and one.
And it depends on the scenario.
You might wanna do one or the other
but basically you want all of these variables to be
quite similar, about the same range of values.
And why is that?
Well, all of these values are going to go
into a neural network where as we'll see just now,
they'll be added up and multiplied by weight,
added up and so on.
And it's just going to be easier for the neural network
to process them, if they're all about the same.
And in fact, you know,
that's just how it is going to be able to work properly.
And if you want to read more about standardization,
normalization and other things that you can do
if you input variables,
a good additional reading paper is called
Efficient BackProp by Yann LeCun 1998.
The links over there.
So Yann LeCun, we're actually going to talk more
about this phenomenal person in the space of deep learning
in the part of the course where we're talking
about convolutional neural networks.
And you'll see that this is definitely a person
who knows what he's talking about.
He's a close friend of Geoffrey Hinton
who we've already seen, who've already mentioned.
So in this paper you'll learn more
about standardization and normalization
but also you can pick up lots of
other different tips and tricks
and he'll be a good source for additional reading
as you go through this course.
So yeah, check it out,
if you are interested in some additional reading.
There we go.
So that's what we need to do with the variables.
And here we've got the output value.
So what can our output value be?
Well, we've got a couple of options.
Output value can be, it can be continuous,
like for instance, price.
It can be binary.
For instance, a person will exit or will stay.
Or it can be a categorical variable.
And if it's a categorical variable
the important thing to remember here is that in that case
your output value won't be just one
it'll be several output values
because these will be your dummy variables
which will be representing your categories.
And that's just how it works.
Just important to remember that in that case
that's how you're going to be getting your categories
out of the artificial neural network.
But let's go back to a simple case of one output value.
And now let's one more point
or a point that we've already made,
I just wanted to reiterate this point.
On the left, you've got a single observation.
So one row from your dataset
and on the right you have a single observation as well.
And that is the same observation.
So important to remember that whatever inputs
you're putting in
that's for one row
and then the output you get that is for that same exact row.
Or if you're training your neural network
then you're putting the inputs in for that one row,
you're putting the output in for that one row.
So if you wanna simplify the complexity,
think of it like a simple linear regression
or a multivariate linear regression.
So you're putting in your values
you have your output,
there's no question about it
when you were talking about things like regression
because we're so used to it.
Same thing here.
It's nothing too complex.
We're just putting in values, we're getting output.
But just remember
that every time it's one row you're dealing with
so you don't get confused
and start putting in like,
thinking that these are different rows
that you're putting into
your artificial neural network or something.
This is all just values in that one row.
So different observation, different characteristics
or attributes relating
to that one observation every single time.
Okay, so next thing that we wanna talk about here
are the synapses?
Is the synapses.
Here we've got synapses
and they all actually get assigned weights.
Weights, we're gonna talk more about weights further down.
But in short, weights are crucial to
artificial neural networks functioning
because weights are how neural networks learn.
By adjusting the weights,
the neural network decides in every single case,
what signal is important,
what signal is not important to certain neuron,
what signal gets passed along
and what signal doesn't get passed along,
or to what strength,
to what extent signals get passed along.
So weights are crucial.
They are the things that get adjusted
through the process of learning.
Like when you're training an artificial neural network,
you're basically adjusting all of the weights
in all the synapses across this whole neural network.
And that's where gradient descent and back propagation
come into play.
And those are concepts that we'll also discuss.
So basically those are the weights.
That's all we need to know for now.
And here we've got the neuron.
So signals go into the neuron
and what happens in the neuron?
So this is the interesting part,
we're talking about the neuron today,
What happens inside the neuron?
So a few things happen.
First thing and the first step is that all of these values
that it's getting, get added up.
So it takes the added, so the weighted sum of all
of the input values that it's getting.
Very simple, right?
It's very, very straightforward.
Just add up, multiply by the weight, add them up.
And then it applies an activation function.
Now we're gonna talk more
about activation functions further down
but it's basically a function that is assigned
to this neuron or to this whole layer
and it is applied to this weighted sum.
And then from that, the neuron understands
if it needs to pass on a signal,
that's the signal that it passes on
that the function applied to the weighted sum.
But basically depending on the function,
the neuron will either pass on a signal it
or it won't pass that signal on.
And that's exactly what happened here in step three,
the neuron passes on that signal
to the next neuron down the line.
And that's what we're going to talk about
in the next tutorial because it is quite an important topic.
We want to delve deeper into the activation function
but hopefully for now,
everything should be pretty clear how,
you know, you've got input values, you've got weights,
you've got these synapses,
you've got something, you know, happens in the neuron,
you've got the weighted sum
and then activation function's applied
and then that is paused down the line.
And that is just repeated
throughout the whole neural network,
on and on and on and on.
You know, thousands, hundreds of thousands of times
depending on how big, how many neurons you have,
how many synapses you have in your neural network.
So there we go.
Hope you enjoyed today's tutorial.
Can't wait to see you next time.
And until then, enjoy deep learning.