In this lecture we are going to look at the models we just used more in depth.
Specifically if you know a little bit about machine learning you may recognize that the models we just
used are called linear regression and logistic regression.
We use logistic regression for classification and we use linear regression for regression.
So this lecture will look at how these models work and also explain why we refer to logistic regression
as a neuron.
Just to remind you of the high level picture here.
Deep Learning is the study of neural networks and neural networks are networks of neurons.
So you can think of these as the basic fundamental unit of computation in deep learning
let's start with linear regression in its most basic form linear regression just means line of best
fit.
As you may recall from your high school math studies a line has the equation y hat equals M X plus b
here X is the input variable M is the slope and B is the y intercept.
When we put these together we get the equation for a line our job of course is to find a line that best
fits our input data
to give you a simple example of how we would use linear regression.
Let's suppose we are trying to predict your salary based on how many years of experience you have.
So in this case x would be the number of years of experience and y would be your salary.
Now you might be wondering what is the interpretation of the slope in y intercept.
In this scenario.
Well let's try to play around with this equation a little bit and see what happens.
We know that b the y intercept is the value of y half when x equals zero.
So in other words if you have zero years of experience this would be your salary.
B If X equals 1 that means you have one year of experience and then your salary would be M Plus B if
x equals two.
That means you have two years of experience and then your salary would be two M A plus B.
In other words M is the increase in salary that you get for each additional year of experience.
Of course in the real world we might want to make a prediction about your salary based on multiple factors
instead of just years of experience.
We might have another input let's say average industry salary.
So now your salary can also depend on what industry you are working in.
So we'll call it years of experience x 1 and we'll call average industry salary x 2 in this scenario.
We would write out our model as Y equals w 1 x 1 plus W2 x 2 plus B in this equation w 1 and W2 are
called the weights of the model and they are essentially the slope for each of the individual inputs
X1 and X2 luckily the interpretation is the same as what we saw previously.
W 1 is the increase in salary when X1 increases by 1 W2 is the increase in salary when x 2 increases
by 1.
B is the salary when both X1 and x 2 or 0
another way to think of the weights is that they tell us how important each input is to predicting the
output.
Imagine the extreme scenario where we have some Ws of IE equals zero.
In this case it doesn't matter how much we increase or decrease X survive the output will not change
at all.
So in other words X Abi has no influence on the output.
That's another way of saying X Abi is irrelevant.
Now imagine that w Saibai is very large in magnitude.
Note that it can be either very positive or very negative.
This means that X Abi has a large influence on the output.
Because a small increase in x Abi leads to a large increase in output that's just another way of saying
that this input feature is very important.
Also the sign of W's supply controls the direction of the influence.
If W Saibai is very positive then an increase in supply will lead to a large increase in the output.
If W Saibai is very negative then an increase in Saibai will lead to a large decrease in the output
now in what way is any of this related to neurons.
Well to understand this you have to understand a little bit about biology.
A neuron is a cell in your brain.
You can think of it as we said before as the fundamental unit of computation neurons can talk to other
neurons using electrical and chemical signals.
So if you picture a neuron you can imagine that there are many other incoming neurons attached to it.
These input terminals are called dendrites.
As you can see in this image those incoming neurons might come from say your eyes or your ears or your
nose or your hands.
So if your eyes see something an electrical signal goes along the nerves in your eyes and travels to
your brain the same thing happens when you hear something or you smell something or you touch something.
Now remember we're picturing a single neuron it's taking in inputs from a lot of different places.
Now this neuron it has to decide if it's going to pass this signal on to outgoing neurons because remember
this neurons kind of in the middle there are some neurons going into it and it's going out to some other
neurons.
Now how does it do that.
Well it does that in a very similar way to linear and logistic regression.
It sums up all the incoming signals and then this summation becomes the outgoing signal that gets passed
on further down the chain.
The next thing you have to know is that not all neuron connections are created equal.
Some connections are strong while others are weak.
Some connections excite the receiving neuron while other connections inhibit the receiving neuron.
This is just like the weights of a regression model.
A large wave either positive or negative has a strong influence on the output.
That's a strong connection.
A small or nearly zero way has only a weak influence on the output so that's a weak connection.
A positive weight influences the output in the positive direction.
So that's like an excited tree neuron in a negative way it influences the output in the negative direction.
So that's like an inhibitory neuron
the signal that gets passed along neurons has a special name.
It's called an action potential.
Basically it's a spike in electrical potential.
So if you measure the electrical potential over time at a particular point in the neuron you would see
a signal like this.
Now action potentials don't behave in a very intuitive way.
In particular you can think of them as binary outcomes just like logistic regression.
In other words the neuron is more like logistic regression than linear regression although the linear
regression equation is the main calculation that takes place in both cases.
So here's how the action potential works.
Basically we're going to sum up all the influences from all the incoming neurons.
If the electrical potential of this sum is greater than some threshold then an action potential will
propagate through the receiving neuron if the electrical potential is less than this threshold then
nothing happens at all.
This is very much like binary classification where we make predictions which are either 0 or 1
in biology.
We call this the all or nothing principle either an action potential fires or doesn't.
As you can see here when the stimulus is too weak there's only a little bump in the response.
So those are not action potentials but once the stimulus reaches a certain threshold of full action
potential spike occurs.
So there's no such thing as a spectrum of action potentials like an action potential of one full two
volts three volts and so forth.
Instead it's just a binary decision.
You get a spike or you don't.
The incoming signals some together are strong enough to generate the next action potential or they will
simply be ignored
so that's how you can think of logistic regression as a neuron.
Each X Abi is an input from some incoming neuron.
Each w survive is a weight that tells us how strong the connection with X Abi is in the sign of WC Abi
tells us whether that connection is excite a tree meaning it positively influences an action potential
or inhibitory meaning it negatively influences an action potential.
The weighted sum of each x Abi which is then added to the bias term or threshold is then passed to the
sigmoid function.
Once the sigmoid function is rounded we get either 1 or 0 telling us whether or not an action potential
should occur.