In this lecture, we are going to discuss forward propagation as it relates to neural networks.
Neural networks are all about making predictions, and so making a prediction in a neural network is
what we refer to as going in the forward direction.
Previously, we looked at logistic regression and how this might be seen as a computer simulation of
a neuron.
Well, let's consider an interesting scenario here.
We have one neuron.
But let's say these same inputs propagate to another output.
Now we have to analyze each of them could be calculating something different given the same inputs.
If you want a sort of Real-World analogy of that, suppose we're looking at a phase one.
Neuron might be looking for the presence of an eye and another might be looking for the presence of
a nose.
In other words, different neurons are finding different features from the input.
Well, how about we add another drone?
Now we have a whole layer of neurons.
The key insight in deep learning made it decades ago by researchers such as Geoffrey Hinton was that
we could stack more neurons on top of the existing neurons.
Remember, that's kind of like how our brain is.
We don't just have one neurons in our brain.
We have many.
So one neuron might be connected to another neuron, which might then be further connected to another
neuron and so forth.
So you end up with this long chain of neurons.
The key idea is that we can think of neurons as uniform, although these days we know a lot about the
brain.
And so this is not physically the case.
Some parts of the brain are different than others, but for our simple computational model, we can
assume the brain is uniform.
There's no difference between a neuron in one part of the brain compared to another part, as you might
expect.
We can just keep doing this.
So we have one layer going to another layer, going to another layer and so on.
So what have we done?
Well, we expanded our concept of neurons in two important ways.
First, we said that the same inputs could be attached to multiple different neurons, each calculating
something different.
This means more neurons per layer.
Second, we said that neurons in one layer could act as inputs to another layer of neurons.
So the first concept is to make the neural network more wide.
The second concept is to make the neural network more deep and believe it or not.
This is all it takes to build a neural network.
Take a neuron.
Add more neurons across the same layer to make it wide and then add more layers to make it deep.
Previously, we looked at the equation for a line and that became our model.
So if you recall that was X plus B, we then said in order to make this into a neuron, we're going
to have multiple inputs and multiple weights.
We're also going to attach a sigmoid at the end in order to map the output to a value between zero and
one, representing something like the probability of an action potential that became the sigmoid of
W transpose x plus b.
The next question to consider is what should we do if we have multiple neurons per layer?
Well, suppose I have some output at the next layer, let's call that Z Sub J to represent the JF neuron
in that layer, then a Z Sub J would just be the sigmoid of W subject transpose x plus a b sub J.
Let's say that J goes from one to M so that we have M neurons in this layer.
That also means we'll have M, WJC and M.
It turns out that we can make this calculation more efficient if we consider Z to be a vector of neuron
values instead of just a scalar.
Then we can write this more compactly.
Now you may find this a little confusing if you're not used to working with matrices and functions of
matrices.
So we have to clarify a few things.
First, Z is a vector of size.
M X is as before a vector of size D or equivalently.
You can think of Z as a column vector of size M by one and X to be a column vector of size D by one.
In order to make this a valid matrix multiplication, we have the following W is a matrix of size D
by M.
So when you transpose it, you get m by B, and then when you multiply that by X, which is of size
D, you get out a vector of size m in order to add the bias term be.
It is also a vector of size m.
A sigmoid here is meant to be an Element Y's operation.
So whatever the size of the input is, it has the same size output.
In other words, it would be as if you apply the sigmoid to each component of its argument individually
and then organize them into a vector or a matrix of the same shape.
Using these concepts.
Let's think about how we might mathematically express going from the input to the output of a neural
network.
First, let's consider that we are going to have multiple layers, so it's not enough to call our weights
just W and B instead of we're going to give them indices.
So let's say w superscript one and B superscript one are the weight and bias of the first layer.
W superscript two and B Superscript two are the weight and bias of the second layer and so forth.
Z superscript one will be the output of the first layer.
Z Superscript two will be the output of the second layer and so forth.
Now, there are two things to notice about the final layer here.
First, you'll notice that I've used the capital letter L to to note the total number of layers.
The letter L is common in deep learning, so if you see the letter L elsewhere, don't be surprised.
Just pay attention to the context.
In this context, we mean the number of layers and of course, the input to the last layer will be the
output of the second last layer.
So we have this nice repeating pattern at each layer.
Each layer is just the same equation over and over again.
So that's what we mean when we say the neural network is uniform.
The second thing to notice is that for now, well, assume that we are still doing binary classification.
Therefore, the final sigmoid maps, the output of the neural network to be a single number between
zero and one.
You can imagine that if we are doing regression, we would not want to have the sigmoid there.
So, of course, this brings us to the next question, what does a neural network for regression look
like?
Surely we are not limited to just binary classification.
And luckily, it's very similar.
All we do is simply remove the final sigmoid.
One interesting thing we can see from this, if we look at only the final layer here is that it appears
just to be regular or linear regression.
And that brings us to our next major point.
This is a very helpful perspective when you're thinking about neural networks.
So I think it's important to learn about early on.
If you think about what we just looked at neural networks for binary classification and neural networks
for regression, they are not that different from what we looked at previously.
If we look at just the final layers, they look just like regular or linear regression and logistic
regression.
That's what we studied in the previous section.
So here's a new way to think about neural networks instead of merely a network of layers of neurons.
Really, we're just doing a series of feature transformations.
So we go from featured a feature to feature.
And then finally, we just have a plain linear regression for regression or logistic regression for
classification.
Using these uniform layers, researchers noticed that neural networks were able to learn hierarchies
of features.
Each layer seemed to be learning something more complex than the last.
And this was the motivation for making neural networks deep.
And that's why this field is called deep learning.
One of my favorite examples of this is with facial recognition that works by visualizing what each layer
has learned.
Researchers found that the initial layers just learn some basic lines of strokes, and in the middle,
the neural networks started to learn about specific features of faces such as eyes, nose, mouth and
so on.
Finally, at the final layers, the neural networks started to see entire faces.
In other words, neural networks are able to break down a problem into smaller sub problems or, in
other words, a hierarchy.