In this lecture we are going to discuss forward propagation as it relates to neural networks.
Neural networks are all about making predictions and so making a prediction in a neural network is what
we refer to as going in the forward direction.
Previously we looked at logistic regression and how this might be seen as a computer simulation of a
neuron.
Well let's consider an interesting scenario.
Here we have one neuron but let's say these same inputs propagate to another output.
Now we have two neurons each of them could be calculating something different given the same inputs.
If you want a sort of real world analogy of that suppose we're looking at a face one neuron might be
looking for the presence of an eye.
And another might be looking for the presence of a nose.
In other words different neurons are finding different features from the input
Well how about we add another neuron.
Now we have a whole layer of neurons
the key insight in deep learning made decades ago by researchers such as Jeffrey Hinton was that we
could stack more neurons on top of the existing neurons.
Remember that's kind of like how our brain is.
We don't just have one neuron in our brain we have many.
So one neuron might be connected to another neuron which might then be further connected to another
neuron and so forth.
So you end up with this long chain of neurons.
The key idea is that we can think of neurons as uniform although these days we know a lot about the
brain.
And so this is not physically the case.
Some parts of the brain are different than others but for our simple computational model we can assume
the brain is uniform.
There is no difference between a neuron in one part of the brain compared to another part as you might
expect.
We can just keep doing this.
So we have one layer going to another layer going to another layer and so on
so what have we done.
Well we expanded our concept of neurons in two important ways.
First we said that the same inputs could be attached to multiple different neurons each calculating
something different.
This means more neurons per layer.
Second we said that neurons in one layer could act as inputs to another layer of neurons.
So the first concept is to make the neuron they work more wide.
The second concept is to make the neural network more deep and believe it or not.
This is all it takes to build a neural network.
Take a neuron add more neurons across the same layer to make it wide and then add more layers to make
it the
previously.
We looked at the equation for a line and that became our model.
So if you recall that was a x plus b we then said in order to make this into a neuron we're going to
have multiple inputs and multiple weights.
We're also going to attach a sigmoid at the end in order to map the output to a value between 0 and
1 representing something like the probability of an action potential.
That became the sigmoid of W transpose x plus B the next question to consider is what should we do if
we have multiple neurons per layer
Well suppose I have some output at the next layer let's call that Z subject to represent the J if neuron
in that layer.
Then Z sub J would just be the sigmoid of W subject transpose x plus a b sub J.
Let's say that J goes from 1 to M so that we have m neurons in this layer.
That also means we'll have m w JS and M B JS
It turns out that we can make this calculation more efficient if we consider Z to be a vector of neuron
values instead of just a scalar then we can write this more compact Lee.
Now you might find this a little confusing if you're not used to working with matrices and functions
of matrices.
So we have to clarify a few things.
First Z is a vector of size m x is as before a vector of size D or equivalently you can think of Z as
a column vector of size m by 1 and x to be a column vector of size V by 1 in order to make this a valid
matrix multiplication.
We have the following.
W is a matrix of size D by m.
So when you transpose it you get m by B and then when you multiply that by X which is of size d you
get out a vector of size m in order to add the bias term B.
It is also a vector of size m the sigmoid here is meant to be an element y's operation so whatever the
size of the input is it has the same size output.
In other words it would be as if you apply the sigmoid to each component of its argument individually
and then organize them into a vector or a matrix of the same shape
using these concepts let's think about how we might mathematically Express going from the input to the
output of a neural network.
First let's consider that we are going to have multiple layers.
So it's not enough to call our weights just w and B instead of we're going to give them indices.
So let's say w superscript 1 and b superscript 1 are the weight and bias of the first layer w superscript
2 and b superscript 2 other weight and bias of the second layer and so forth.
Ze superscript 1 will be the output of the first layer Zi superscript 2 will be the output of the second
layer and so forth.
Now there are two things to notice about the final layer here.
First you'll notice that I've used the capital letter L to denote the total number of layers the letter
L is common and deep learning.
So if you see the letter L elsewhere don't be surprised.
Just pay attention to the context in this context we mean the number of layers and of course the input
to the last layer will be the output of the second last layer.
So we have this nice repeating pattern at each layer each layer is just the same equation over and over
again.
So that's what we mean when we say that none that work is uniform.
The second thing to notice is that for now we'll assume that we are still doing binary classification.
Therefore the final sigmoid map's the output of the neural network to be a single number between 0 and
1.
You can imagine that if we are doing regression we would not want to have the sigmoid there
so of course that this brings us to the next question.
What does a neuron that work for regression look like.
Surely we are not limited to just binary classification and luckily it's very similar.
All we do is simply remove the final sigmoid.
One interesting thing we can see from this if we look at only the final layer here is that it appears
just to be regular or linear regression.
And that brings us to our next major point.
This is a very helpful perspective when you're thinking about neuron networks.
So I think it's important to learn about early on.
If you think about what we just looked at neuron that works for binary classification and neural networks
for regression they are not that different from what we looked at previously.
If we look at just the final layers they look just like regular or linear regression and logistic regression.
That's what we studied in the previous session.
So here's a new way to think about neural networks instead of merely a network of layers of neurons.
Really we're just doing a series of feature transformations so we go from feature to feature to feature.
And then finally we just have a plain linear regression for regression or logistic regression for classification
using these uniform layers.
Researchers noticed that neural networks were able to learn hierarchies of features each layer seemed
to be learning something more complex than the last.
And this was the motivation for making neural networks deep.
And that's why this field is called deep learning one of my favorite examples of this is with facial
recognition that works by visualizing what each layer has learned.
Researchers found that the initial layers just learned some basic lines and strokes and in the middle
the known that work started to learn about specific features of faces such as eyes nose mouth and so
on.
Finally at the final layers the neural networks started to see entire faces.
In other words neural networks are able to break down a problem into smaller sub problems or in other
words a hierarchy.