In this lecture, we are going to discuss forward propagation as it relates to neural networks, neural
networks are all about making predictions and so making a prediction in a neural network is what we
refer to as going in the forward direction.
Previously, we looked at logistic regression and how this might be seen as a computer simulation of
a neuron.
Well, let's consider an interesting scenario here.
We have one neuron, but let's say these same inputs propagate to another output.
Now we have two neurons.
Each of them could be calculating something different, given the same inputs if you want a sort of
real world analogy of that.
Suppose we're looking at a phase one neuron.
Might be looking for the presence of an eye and another might be looking for the presence of a nose.
In other words, different neurons are finding different features from the input.
Well, how about we add another neuron now we have a whole layer of neurons.
The key insight in deep learning made decades ago by researchers such as Geoffrey Hinton was that we
could stack more neurons on top of the existing neurons.
Remember, that's kind of like how our brain is.
We don't just have one neuron in our brain, we have many.
So one neuron might be connected to another neuron, which might then be further connected to another
neuron and so forth.
So you end up with this long chain of neurons.
The key idea is that we can think of neurons as uniform, although these days we know a lot about the
brain.
And so this is not physically the case.
Some parts of the brain are different than others, but for our simple computational model, we can
assume the brain is uniform.
There's no difference between a neuron in one part of the brain compared to another part.
As you might expect, we can just keep doing this so we have one layer going to another layer, going
to another layer and so on.
So what have we done?
Well, we expanded our concept of neurons in two important ways.
First, we said that the same inputs could be attached to multiple different neurons, each calculating
something different.
This means more neurons per layer.
Second, we said that neurons in one layer could act as inputs to another layer of neurons.
So the first concept is to make the neural network more wide.
The second concept is to make the neural network more deep.
And believe it or not.
This is all it takes to build a neural network.
Take a neuron, add more neurons across the same layer to make it wide, and then add more layers to
make a deep.
Previously, we looked at the equation for a line and that became our model.
So if you recall, that was A plus B, we then said in order to make this into a neuron, we're going
to have multiple inputs and multiple weights.
We're also going to attach a sigmoid at the end in order to map the output to a value between zero and
one representing something like the probability of an action potential that became the sigmoid of transpose
X plus B.
The next question to consider is, what shall we do if we have multiple neurons per layer?
Well, suppose I have some output at the next layer.
Let's call that subject to represent the JF neuron in that layer, then Z subject would just be the
sigmoid of W's subject, transpose X plus a B subject.
Let's say that J goes from one to M. So that we have M neurons in this layer.
That also means we'll have M W.J. and veejays.
It turns out that we can make this calculation more efficient if we consider it to be a vector of neuron
values instead of just a scalar, then we can write this more compactly.
Now, you might find this a little confusing if you're not used to working with matrices and functions
of matrices.
So we have to clarify a few things.
First, Z is a vector of size, M X is, as before, a vector of size D or equivalently, you can think
of Z as a column vector of size and by one and X to be a column vector of size DBI one.
In order to make this a valid matrix multiplication, we have the following W is a matrix of size D
by M.
So when you transpose it you get M by B and then when you multiply that by X, which is of size D,
you get out a vector of size M in order to add the bias term B, it is also a vector of size M, a sigmoid
here is meant to be an element y's operation.
So whatever the size of the input is, it has the same size output.
In other words, it would be as if you apply the sigmoid to each component of its argument individually
and then organize them into a vector or matrix of the same shape.
Using these concepts, let's think about how we might mathematically express going from the input to
the output of a neural network.
First, let's consider that we are going to have multiple layers so it's not enough to call our weights
just WNBA.
Instead, we're going to give them indices.
So let's say W superscripts one and be superscripts one are the weight and bias of the first layer.
W superscript two and B superscript two are the weight and bias of the second layer and so forth.
Z superscript.
One will be the output of the first layer, z superscript two will be the output of the second layer
and so forth.
Now, there are two things to notice about the final layer here.
First, you'll notice that I've used the capital letter L to denote the total number of layers.
The letter L. is common and deep learning.
So if you see the letter L. elsewhere, don't be surprised.
Just pay attention to the context.
In this context, we mean the number of layers.
And of course, the input to the last layer will be the output of the second last layer.
So we have this nice repeating pattern at each layer.
Each layer is just the same equation over and over again.
So that's what we mean when we say the neural network is uniform.
The second thing to notice is that for now, we'll assume that we are still doing binary classification.
Therefore, the final sigmoid maps, the output of the neural network to be a single number between
a zero and one.
You can imagine that if we are doing regression, we would not want to have the sigmoid there.
So, of course, this brings us to the next question what does a neuron that work for regression look
like?
Surely we are not limited to just binary classification and luckily it's very similar.
All we do is simply remove the final sigmoid.
One interesting thing we can see from this, if we look at only the final layer here is that it appears
just to be regular old linear regression.
And that brings us to our next major point, this is a very helpful perspective when you're thinking
about neural networks.
So I think it's important to learn about early on, if you think about what we just looked at, neural
networks for binary classification and neural networks for regression, they are not that different
from what we looked at previously.
If we look at just the final layers, they look just like regular old linear regression and logistic
regression.
That's what we studied in the previous section.
So here's a new way to think about neural networks instead of merely a network of layers of neurons.
Really, we're just doing a series of feature transformations.
So we go from feature to feature to feature.
And then finally, we just have a plain linear regression for regression or logistic regression for
classification.
Using these uniform layers, researchers noticed that neural networks were able to learn hierarchies
of features, each layer seemed to be learning something more complex than the last.
And this was the motivation for making neural networks deep.
And that's why this field is called deep learning.
One of my favorite examples of this is with facial recognition that works by visualizing what each layer
has learned.
Researchers found that the initial layers just learned some basic lines and strokes.
And in the middle, the known networks started to learn about specific features of faces such as eyes,
nose, mouth and so on.
Finally, at the final layers, the neural networks started to see entire faces.
In other words, neural networks are able to break down a problem into smaller sub problems or, in
other words, a hierarchy.