So in this video, we are going to discuss logistic regression.
Now, despite its name, logistic regression is actually used as a classifier.
It can be thought of as the classification analogue of linear regression.
So how does it work?
Well, the question you have to answer is, why can't we just use linear regression as it is?
For example, applying this rule, all data is the same.
You would simply have targets which are made up of zeros and ones.
For example, instead of trying to predict a student's exam grade, you simply want to predict whether
they passed or failed.
So why can't we use linear regression for this task?
Well, the answer is that technically you can, but it's not ideal.
Now, this is outside the scope of this course, but you're encouraged to check extra reading text if
you want to learn more.
Basically, linear regression assumes that your prediction errors are Gaussian.
If that is not the case, then you violated the assumptions of the model.
Instead, when your targets are binary, it's more correct.
You base your error on the binary distribution more commonly known as the Bernoulli distribution.
Now, again, we won't go through any detail since this is just for intuition, but basically this is
the distribution you would use to model a coin toss.
There's just one piece of math you need to know about, which is how we go from a linear model to predicting
a category.
The answer is the logistic function, also known as the sigmoid.
It has this nice formula you can see here.
But the important thing is to understand its shape.
As you can see, it has this shape with asymptote on the two ends.
If its input goes to infinity, the output of the logistic function approaches one.
If its input goes to minus infinity, the output of the logistic function approaches zero in the center.
When its input is zero.
The output is zero point five.
Thus, this is a reasonable function for mapping real numbers to probabilities.
As mentioned previously, you really want to understand this pattern, W, transpose X, plus B, remember
that intuitively this forms a line or a hydroplane.
Well, this is going to be the input to our logistic function.
So basically, logistic regression is like linear regression, except we have this one extra step where
we pass the result of W, transpose X plus B through the sigmoid, and that gives us a probability between
zero and one, which makes sense if you want to predict a coin toss.
In fact, this output tells us the probability that the target is equal to one.
So we write it as P of Y equals one given X..
Our model tells us what the probability of Y being one is, given the input X, for example, given
that Alice studied for two hours and slept for eight hours, her probability of passing the exam is
80 percent.
Now, in order to make a decision, we must still pick either zero or one as a prediction.
In this case, we would have round the probability, which gives us either zero or one.
Sometimes people like to choose their own threshold, but we won't discuss that in this course.
So, as you know, one of my important rules for machine learning is machine learning is nothing but
geometry.
So how does that work for classification and specifically logistic regression?
Well, it helps to think of the extreme cases.
As you know, W transpose X plus B is the equation for a line or a plane.
So if you transpose X plus B is equal to zero, that means our data point lies directly on the line.
As you recall, when we plug zero into the sigmoid, we get zero point five.
This makes sense since if we are directly on the line, then the probability that Y belongs to either
class would be 50 percent.
Now imagine that X is very far away from the line so that W transpose X plus B approaches infinity.
In this case, the probability is going to approach zero or one.
This also makes sense because as you move further and further away from the line, you are more certain
that the data point belongs to either of the two classes.
So as a final point for this lecture, we need to discuss the concept of multi class problems.
Now, at this point, we won't go too in-depth, since I want to save that for some later time.
But I want to mention that it is possible to extend our linear model to predict multiple classes.
This is similar to the idea that we can have linear regression with multiple output targets.
The basic idea is you're still going to get a probability, but instead of being from a Bernoulli distribution,
it's going to be from a categorical distribution.
Basically, a categorical distribution for categories can be described by different probabilities.
For example, imagine rolling a die.
In this case, K equals six, so you'd have the probability of rolling a one or two or three and so
forth.
The only requirement, of course, is that those probabilities have to sum to one.
The only important thing to mention is how this multiclass output works in code, so typically you're
going to make a prediction on an input vectors.
At the same time, as you recall, you're going to pass in an input matrix X of shape and build the
output of this model will be another matrix of size.
And becae that is to say, for each of the samples you'll have key output probabilities, which makes
sense in light of what we just discussed.
The important thing to notice is that these probabilities are along the roads, so each row must come
to one.
Now, in order to decide which class each sample belongs to, it doesn't make sense to round as it does
in the binary case.
Imagine, for example, you had 10 classes and each of them had an output probability of zero point
one.
If you rounded them, you would round each of them to zero, which obviously doesn't make any sense.
In fact, if you rounded K numbers, you'd still have key numbers, which also doesn't make any sense.
So in the multiclass case, the proper thing to do is take the Amax.
That is to say, you pick the class, which is the highest probability, and you do this for each row
to get one class for each of the samples.