In this lecture, we're going to look at the models we just used more in-depth.
Specifically, if you know a little bit about machine learning, you may recognize that the models we
just used are called linear regression and logistic regression.
We use logistic regression for classification.
And we use linear regression for regression.
So this lecture will look at how these models work and also explain why we refer to logistic regression
as a neuron.
Just to remind you of the high level picture here, deep learning is the study of neural networks and
neural networks are networks of neurons.
So you can think of these as the basic fundamental unit of computation in deep learning.
Let's start with linear regression in its most basic form, linear regression just means line of best
fit.
As you may recall from your high school math studies, a line has the equation Y hat equals m x plus
b.
Here is the input variable M is the slope, and B is the Y intercepts when we put these together, we
get the equation for a line.
Our job, of course, is to find a line that best fits our input data.
The next lecture will cover exactly how to do that.
But in this lecture, it will be concerned with the form of the model itself.
To give you a simple example of how we would use linear regression.
Let's suppose we are trying to predict your salary based on how many years of experience you have.
So in this case, X would be the number of years of experience and Y would be your salary.
Now you might be wondering what is the interpretation of the slope and y intercept in this scenario?
Well, let's try to play around with this equation a little bit and see what happens.
We know that B the Y intercept is the value of Y hat when X equals zero.
So in other words, if you have zero years of experience, this would be your salary.
B, If X equals one, that means you have one year of experience and then your salary would be M plus
b if X equals two.
That means you have two years of experience and then your salary would be two m plus B.
In other words, M is the increase in salary that you get for each additional year of experience.
Of course, in the real world, we might want to make a prediction about your salary based on multiple
factors instead of just years of experience, we might have another input, let's say average industry
salary.
So now your salary can also depend on what industry you are working in.
So we'll call it years of experience x one and we'll call average industry salary x two.
In this scenario, we would write out our model as Y equals one x one plus w two x two plus B.
In this equation, W1 and W2 are called the weights of the model, and they are essentially the slope
for each of the individual inputs X, X1 and X.
Luckily, the interpretation is the same as what we saw previously.
W1 is the increase in salary when X1 increases by one W2 is the increase in salary when X two increases
by one.
B is the salary when both X1 and X two zero.
Another way to think of the Waits is that they tell us how important each input is to predicting output.
Imagine the extreme scenario where we have some use of AI equals zero.
In this case, it doesn't matter how much we increase or decrease exabyte, the output will not change
at all.
So in other words, SBY has no influence on the output.
That's another way of saying exabyte is irrelevant.
Now, imagine that use of AI is very large in magnitude.
Note that it can be either very positive or very negative.
This means that exabyte has a large influence on the output.
Because a small increase in ZABI leads to a large increase in the output.
That's just another way of saying that this input feature is very important.
Also, the sign of Saibai controls the direction of the influence, if WCB is very positive that an
increase in exabyte will lead to a large increase in the output.
If you submit is very negative, then an increase in SBY will lead to a large decrease in the output.
Now, in what way is any of this related to neurons?
Well, to understand this, you have to understand a little bit about biology.
A neuron is a cell in your brain.
You can think of it, as we said before, as the fundamental unit of computation.
Neurons can talk to other neurons using electrical and chemical signals.
So if you picture a neuron, you can imagine that there are many other incoming neurons attached to
it.
These input terminals are called dendrites, as you can see in this image.
Those incoming neurons might come from, say, your eyes or your ears or your nose or your hands, so
if your eyes see something, an electrical signal goes along the nerves in your eyes and travels to
your brain.
The same thing happens when you hear something or you smell something or you touch something.
Now, remember, we're picturing a single neuron.
It's taking in inputs from a lot of different places.
Now, this neuron, it has to decide if it's going to pass this signal on to outgoing neurons.
Because remember this neurons kind of in the middle, there are some neurons going into it and it's
going out to some other neurons.
Now how does it do that?
Well, it does that in a very similar way to linear and logistic regression.
It sums up all the incoming signals.
And then this summation becomes the outgoing signal that gets passed on further down the chain.
The next thing you have to know is that not all neuron connections are created equal.
Some connections are strong, while others are weak.
Some connections excite the receiving neuron, while other connections inhibit the receiving neuron.
This is just like the weights of a regression model.
A large wave, either positive or negative, has a strong influence on the output.
That's a strong connection.
A small or nearly zero way has only a weak influence on the output.
So that's a weak connection in a positive weight influences the output in the positive direction.
So that's like an excitatory neuron in a negative way.
It influences the output in the negative direction.
So that's like an inhibitory neuron.
The signal that gets passed along neurons has a special name.
It's called an action potential.
Basically, it's a spike in electrical potential.
So if you measure the electrical potential over time at a particular point in the neuron, you would
see a signal like this.
Now, action potentials don't behave in a very intuitive way.
In particular, you can think of them as binary outcomes, just like logistic regression.
In other words, the neuron is more like logistic regression than linear regression, although the linear
regression equation is the main calculation that takes place in both cases.
So here's how the action potential works.
Basically, we're going to sum up all the influences from all the incoming neurons.
If the electrical potential of this sum is greater than some threshold than an action potential will
propagate through the receiving neuron.
If the electrical potential is less than this threshold, then nothing happens at all.
This is very much like binary classification, where we make predictions which are either zero or one.
In biology, we call this the all or nothing principle, either an action potential fires or it doesn't.
As you can see here, when the stimulus is too weak, there's only a little bump in the response.
So those are not action potentials.
But once the stimulus reaches a certain threshold, a full action potential spike occurs.
So there's no such thing as a spectrum of action potentials, like an action potential of one volt two
volts, three volts and so forth.
Instead, it's just a binary decision.
You get a spike or you don't the incoming signals.
Some together are strong enough to generate the next action potential, or they will simply be ignored.
So that's how you can think of logistic regression as a neuron, each Zoabi is an input from some incoming
neuron.
Each WCB is a weight that tells us how strong the connection with exabyte is and the sign of W. Saibai
tells us whether that connection is excitatory, meaning it positively influences an action potential
or inhibitory, meaning it negatively influences an action potential.
The weighted sum of each x ABI, which is then added to the bias term or threshold, is then passed
through the sigmoid function.
Once the sigmoid function is rounded, we get either one or zero telling us whether or not an action
potential should occur.