In this lecture, we are going to look at the linear models we use previously more in depth, specifically,
our final goal is that we want to know how linear regression and logistic regression make up the building
blocks of a neural network.
You've seen that we can use logistic regression for classification and we use linear regression for
regression.
So this lecture will review how those models work and also explain why we refer to logistic regression
as a neuron.
And just to remind you of the high level picture here, deep learning is the study of neural networks
and neural networks are networks of neurons.
So you can think of these as the basic fundamental unit of computation in ziplining.
Let's start with linear regression in its most basic form, linear regression just means line of best
fit.
As you may recall from your high school math studies, a line has the equation.
Y hat equals M X plus B.
Here is the input variable, M is the slope, and B is the Y intercept.
When we put these together, we get the equation for a line.
Our job, of course, is to find a line that best fits our input data.
The next lecture will cover exactly how to do that, but in this lecture it will be concerned with the
form of the model itself.
To give you a simple example of how we would use linear regression, let's suppose we are trying to
predict your salary based on how many years of experience you have.
So in this case, X would be the number of years of experience and Y would be your salary.
Now, you might be wondering what is the interpretation of the slope and y intercept in this scenario?
Well, let's try to play around with this equation a little bit and see what happens.
We know that B, the Y intercept is the value of Y hat when X equals zero.
So in other words, if you have zero years of experience, this would be your salary.
B, if X equals one, that means you have one year of experience and then your salary would be M plus
B if X equals two, that means you have two years of experience and then your salary would be two M,
A plus B..
In other words, M is the increase in salary that you get for each additional year of experience.
Of course, in the real world, we might want to make a prediction about your salary based on multiple
factors instead of just years of experience, we might have another input, let's say average industry
salary.
So now your salary can also depend on what industry you are working in.
So we'll call it years of experience one and we'll call average industry salary X two.
In this scenario, we would write out our model as Y equals one X one plus W two X two plus B.
In this equation, one and two are called the weights of the model, and they are essentially the slope
for each of the individual inputs, X1 and x2.
Luckily, the interpretation is the same as what we saw previously.
One is the increase in salary when X one increases by one, two is the increase in salary when X two
increases by one, B is the salary when both X one and two are zero.
Another way to think of the waits is that they tell us how important each input is to predicting the
output.
Imagine the extreme scenario where we have some WSI equals zero.
In this case, it doesn't matter how much we increase or decrease Zevi, the output will not change
at all.
So in other words, Zibi has no influence on the output.
That's another way of saying Zibi is irrelevant.
Now imagine that Sabai is very large in magnitude, not that it can be either very positive or very
negative.
This means that Zibi has a large influence on the output.
Because a small increase in Zibi leads to a large increase in the output.
That's just another way of saying that this input feature is very important.
Also, the sign of WSI controls the direction of the influence if WSI is very positive that an increase
in Zibi will lead to a large increase in the output, if Sehbai is very negative, that an increase
in Zibi will lead to a large decrease in the output.
Now, in what way is any of this related to neurons?
Well, to understand this, you have to understand a little bit about biology.
A neuron is a cell in your brain.
You can think of it, as we said before, as the fundamental unit of computation neurons can talk to
other neurons using electrical and chemical signals.
So if you picture a neuron, you can imagine that there are many other incoming neurons attached to
it.
These input terminals are called dendrites, as you can see in this image.
Those incoming neurons might come from, say, your eyes or your ears or your nose or your hands.
So if your eyes see something, an electrical signal goes along the nerves in your eyes and travels
to your brain.
The same thing happens when you hear something or you smell something or you touch something.
Now, remember, we're picturing a single neuron.
It's taking in inputs from a lot of different places.
Now, this neuron, it has to decide if it's going to pass this signal on to outgoing neurons.
Because remember this neuron's kind of in the middle, there are some neurons going into it and it's
going out to some other neurons.
Now, how does it do that?
Well, it does that in a very similar way to linear and logistic regression, it sums up all the incoming
signals and then this summation becomes the outgoing signal that gets passed on further down the chain.
The next thing you have to know is that not all neuron connections are created equal, some connections
are strong, while others are weak.
Some connections excite the receiving neuron while other connections inhibit the receiving neuron.
This is just like the weights of a regression model.
A large wave, either positive or negative, has a strong influence on the output.
That's a strong connection.
A small or nearly zero weight has only a weak influence on the output.
So that's a weak connection, a positive way.
It influences the output in the positive direction.
So that's like an excitatory neuron in a negative way.
It influences the output in the negative direction.
So that's like an inhibitory neuron.
The signal that gets passed along Neurons has a special name, it's called an action potential.
Basically, it's a spike in electrical potential.
So if you measure the electrical potential over time at a particular point in the neuron, you would
see a signal like this.
Now, action potentials don't behave in a very intuitive way.
In particular, you can think of them as binary outcomes, just like logistic regression.
In other words, the neuron is more like logistic regression than linear regression, although the linear
regression equation is the main calculation that takes place in both cases.
So here's how the action potential works.
Basically, we're going to sum up all the influences from all the incoming neurons.
If the electrical potential of this sum is greater than some threshold than an action, potential will
propagate through the receiving neuron.
If the electrical potential is less than this threshold, then nothing happens at all.
This is very much like binary classification where we make predictions which are either zero or one.
In biology, we call this the all or nothing principle, either an action, potential fires or it doesn't.
As you can see here, when the stimulus is too weak, there's only a little bump in the response.
So those are not action potentials.
But once the stimulus reaches a certain threshold, a full action potential spike occurs.
So there's no such thing as a spectrum of action potentials like in action, potential of one or two
volts, three volts and so forth.
Instead, it's just a binary decision.
You get a spike or you don't.
The incoming signals, some together are strong enough to generate the next action potential or they
will simply be ignored.
So that's how you can think of logistic regression as a neuron each Zibi is an input from some incoming
neuron.
Each WSIB is a weight that tells us how strong the connection with Zibi is.
And the sign of Sabai tells us whether that connection is excitatory, meaning it positively influences
an action potential or inhibitory, meaning it negatively influences an action potential.
The weighted sum of each Zibi, which is then added to the bias term or threshold, is then passed through
the sigmoid function.
Once the sigmoid function is rounded, we get either one or zero telling us whether or not an action
potential should occur.