So in this lecture, we will be continuing our discussion about the intuition behind logistic regression.
Now one question you may have at this point is how do we find the weights in the biased terms in the
first place?
This question is not a question will be answering at this time, but we will eventually.
For now, the important thing to pay attention to is the practical side, which is how do we do it in
Python?
Well, you're in luck because in addition to my rule, all data is the same.
You now get to learn one of my other rules, which is all machine learning interfaces are the same.
In particular, if you have used any classifier inside you learn which you have, then you already know
how to do this.
In particular, we begin by creating an instance of the logistic regression class.
The next step is to call the fit function, which will train our model based on the data set we pass
in.
Note that training logistic regression is an iterative process.
So one argument to the constructor is Max Ed, which controls the maximum number of iterations when
you call fit.
For some data sets, you may need more iterations than the default.
You really just have to test it out to see what's right for your particular data set.
After we've trained our model, we can use the model to make predictions for new data.
There are two ways to do this.
The first way is to call models that predict which returns and in length, one dimensional array of
predicted labels.
Note that if you have distinct classes, then this will contain integers from zero up to K minus one
inclusive.
The second way is to call Model Duck, predicts Prabha, which returns a matrix of posterior probabilities.
If you have any data points and classes, then the output will be a two dimensional array of size n
by K the value at the end throw in case the column is the probability that the input belongs to Class
K.
As you recall, probabilities might be useful in certain scenarios like computing the AUC.
So now that you understand how to train a logistic regression model in practice, it's time to return
to the intuition in particular after we found all the weights and vice terms.
Is it possible to interpret what they mean?
The answer is yes.
This is one of the best reasons to use a model like logistic regression.
Not only is it very powerful, it is very easy to interpret, and these explanations will please your
clients and stakeholders.
So to understand how to interpret the model weights, we're going to start with an assumption.
This assumption is that all the input features of X are non-negative that is greater than or equal to
zero.
This makes sense in the context of a field such as in Ielpi, where X might represent word counts.
Obviously, word counts must always be greater than or equal to zero.
If this is not the case, then the ideas I'm about to describe still apply, but they're just reversed
when the inputs are negative.
So suppose that our model has three inputs, which are BMI, exercise frequency and hair length.
The output is whether or not the patient will experience a severe reaction to COVID.
Now, if these input features seem a bit weird, don't worry as we'll be explaining them shortly.
It may help to draw a schematic of the logistic regression model.
We draw the three inputs the circles in the weights as lines connecting the circles to a summation note
after going through the summation note.
We then apply the sigmoid function, which gives us the output p of Y equals one given X.
So suppose that the weight for BMI is very large and positive, if you need a concrete value, let's
suppose it's too.
If this is the case, then the BMI multiplied by this weight will be even more positive, which will
make the activation larger.
In fact, multiplying BMI by two will obviously increase it by a factor of two.
Now, as you recall, a large activation pushes the probability closer to one, therefore, we can interpret
large positive ways to mean that the corresponding feature as a large positive influence on the output
being one.
Now, let's look at the second input, which is exercise frequency, suppose that the weight for this
input is small and negative.
Suppose it's minus 0.5.
So what effect does this have?
Well, since it's negative, then after multiplying by a positive input feature, it brings the activation
down and activation going to the left or towards negative infinity.
Results in the output being closer to zero, as you recall.
However, since the magnitude of this weight is small, it has less of an effect compared to a weight
which is large.
So this is a lesson that there are two factors that matter the magnitude of the weight and the sign.
If the sign is positive, then it brings the output closer to one.
If the sign is negative, then it brings the output closer to zero.
So a positive weight tells us that a larger value of the input feature is correlated with the positive
class.
A negative weight tells us that a larger value of the input feature is correlated with the negative
class if the magnitude is larger than it has a more pronounced effect.
So a weight with a larger magnitude tells us that this input matters a lot in predicting the output.
A weight with a small magnitude tells us that this input doesn't matter as much.
Finally, let's consider the third input, which is hair length.
Obviously, this has no effect on the severity of COVID because you could simply cut your hair and thus
its weight would be zero.
So in practice, if you find a way that is zero or very close to zero, then the interpretation is that
this input is irrelevant.
Of course, this is simply an extension of the fact that the magnitude of the weight matters.
So now that we've discussed how to interpret the weights in the binary case, let's consider the multiclass
case.
Luckily, the interpretation is largely the same.
It just seems a bit more complicated because we have more weights and they sort of interact with each
other due to the soft max.
The best way to interpret the weights is to consider them as a matrix.
Suppose that our model has the inputs and outputs instead of doing separate products with separate weight
vectors of size d.
It's simpler to do a single matrix.
Multiply with a weight matrix of size d by K.
So if X is a vector of size D and we multiply that with a matrix of size D by K, we will get a vector
of size K, which represents the K activations.
So let's call this weight matrix big w.
Now let's consider the case where the value in a big W at a D column K is large.
If this value is large and positive, then it will increase the K ith activation.
Therefore, the interpretation is that input feature D has a positive correlation with the K of class
being the target.
Similarly, if food K is large and negative, then input feature D has a negative correlation with the
K of class being the target.
And again, magnitude matters in pretty much the same way.
If the magnitude is small, then this input feature has less of an effect.
If the magnitude is large, then this input feature has more of an effect.
If the magnitude is zero, then the input feature has no effect.