So in this lecture, we will be discussing the intuition behind logistic regression, which is the machine
learning model will be using to do sentiment analysis.
Now one interesting way to compare what we are learning now to what we previously learned, meaning
niveis is by thinking about these techniques in terms of the high level topics in this course.
As you recall, this course started out with two major sections that were very distinct.
One of them was vector models, where we thought about NLP in terms of converting text into vectors.
The other one was probability models where we thought of NLP in terms of Markov chains and other related
methods.
So pretty obviously naive Bayes is a model based on probability.
In this section, you will see that linear classification of which logistic regression is an example
is based on vectors instead of probabilities.
So this is a nice way to tie in what we learned earlier in this course.
Let's now review the vector perspective on the classification task.
As you recall, one of the nice consequences of converting text into vectors is that this allows us
to plot them on a grid, of course in practice.
These will be very high dimensional grids, but for the purpose of this visualization, we will pretend
they are two dimensional grids.
So the beauty of this method is that it makes understanding classification a simple problem of geometry.
As you recall, our job in building a so-called classifier is to simply find a line or a curve that
can separate points of different colors.
Remember that different colors represent different classes.
So one color might represent positive sentiment, while another might represent negative sentiment and
so forth.
Note that this approach applies to both binary and multi class problems in the multiclass case.
It's simply that we will have more than two colors, but the general idea remains the same.
We want to separate colors by either a line or a curve.
Of course, in higher dimensions, a line becomes a plane or a hybrid plane, while the curve becomes
a surface or hyper surface.
Note that logistic regression is a linear model, and so the separating boundary will be a hyper plane
instead of a hybrid surface.
Now, in order to really understand logistic regression, it is necessary to use a little bit of math.
Luckily, if you just want to understand the intuition, it doesn't require anything beyond approximately
tenth grade mathematics.
So that being said, let's review the equation for a line.
As you recall, the equation for a line is Y equals m x plus b.
In this form, M represents the slope, while B represents the Y intercept.
So hopefully this is just a review.
But if you don't understand this, and please let me know on the Q&A.
Now for machine learning, it is better to use different symbols for our line in particular.
Recall that each axis on this grid represents a component of our feature vector, which we call X.
Furthermore, in machine learning, we typically reserve the letter Y for the output or target of our
model, and thus it's not a good symbol to use on this grid.
So because our feature vector is called X, then we'll simply refer to the components of X as X1 and
X2, as you recall.
This is because our grid currently has two dimensions.
Of course, this is pretty trivial.
Our line now has the equation x two equals m times x one a plus b, so hopefully not very challenging
so far.
The next step is to represent our line in a different form.
In particular, it's much more convenient to have it in the form of W one times x one plus W two times
X two plus B equals zero as an exercise.
If this looks unfamiliar to you, I would recommend trying to convert this equation for a line into
the format we previously saw.
Basically, your goal is to isolate X two by moving all the other variables to the other side.
After doing some algebra, you should get that X two is equal to minus W one over two times x one plus
minus B over W two.
So you might want to try that yourself to confirm that this is correct.
What this tells us is that the slope is minus W one over W2 and the intercept is minus B over W2.
Note that this new B is not the same as the B we saw previously.
It's simply that B is a common letter used in both these scenarios.
In any case, the important point to draw from this is that both of these forms are valid representations
of a line.
So in machine learning, there is some terminology you should know.
In particular, we call the use of weights and we call the B term the bias.
So this is the terminology will be using throughout the course, especially when it comes to deep learning
and neural networks.
As you'll soon see, logistic regression can be thought of as a neuron, which is the basic building
block of neural networks.
Here's another point to consider.
As you recall, the whole reason we have a line is because we want to use the line to figure out which
side a new data point falls on.
For example, if it falls on one side, we say it's spam.
Otherwise, we say it's not spam.
Now you might at first say, Well, why can't I just look at this graph?
And there are two reasons why that won't work.
Firstly is that looking at a graph is not automatic.
Remember, the whole point of this is to do everything inside a computer.
So the process can't involve you looking at a graph.
Secondly, recall that in practice, we're going to have many dimensions, maybe even thousands.
Therefore, our line will in fact be a high dimensional hydroplane, which you cannot see.
And thus, the only way to do it is by doing a calculation inside your computer.
So let's take a very simple line X1 plus X2 minus one equals zero.
In this case, both of our weights W1 and W2 are just one, and the bias term is minus one.
The main thing we want to consider is the expression on the left side that is X1 plus X2 minus one when
this is equal to zero.
It's the equation of our line.
But now this expression by itself is our main object of interest.
So let's see what happens if we plug in the vector X1 equals one and X2 equals zero.
In this case, the point falls on the line, which makes sense since when the left side expression is
equal to zero, that is the equation of our line.
Now let's plug in the point X1 equals zero.
X2 equals zero.
In this case, we get minus one, which falls to the left side of the line.
Now, let's plug in the point one equals one and X two equals one.
In this case, we get plus one, which falls to the right side of the line.
Basically, here's the pattern you should notice when this expression is equal to zero, that's the
line itself.
So any point that satisfies the quality will be on the line.
When this expression is less than zero, then the point we plugged in falls to the left of the line
when this expression is greater than zero.
Then the point we plugged in falls to the right of the line.
I encourage you to try out different points to convince yourself that this pattern holds for all points.
Note that we can simplify the rule we just came up with by assigning the expression we learned to a
new name of X h stands for activation.
So our rule can be stated as air of X is equal to zero, then X is on the line when they have X is greater
than zero.
We are on one side of the line.
When they have X is less than zero.
We are on the other side of the line.
One thing to pay attention to is that the equation for a line is not unique in the format we've presented.
For example, take the line X1 plus X2 minus one equals zero.
I encourage you to convince yourself that two times X1 plus two times x two minus two equals zero represents
the exact same line.
Furthermore, we can even negate all the values.
So minus x one minus x two plus one equals zero still represents the same line.
In other words, an infinite collection of weights and biases can represent the exact same boundary.
This is important since this means that a Vex being greater than zero doesn't automatically imply that
X will be to the right or to the left of some line.
Instead, it depends on the sign in front of you and B, for example, if we plug in one one into our
previous expression, we would get plus one.
But if we plug it into the negated expression, we would get minus one.
And remember, these represent the exact same line.
Now, as you recall, we've been looking at a two dimensional grid since, that's all we can practically
see.
However, in practice, our data sets will have many more dimensions.
In this case, it becomes infeasible to write down W one x one plus W two x two plus W three x three
and so forth.
Instead, what we actually do is use vector notation.
As you recall, X is actually a vector.
Suppose X is a vector of size D since there must be a component of W for every component of X.
This means that W is also a vector of size D.
Furthermore, you may recognize that this Element Y's product in summation is in fact what is known
as a dot product.
Therefore, a much more compact way to write our expression is W Transpose X plus b.
Now, recall that in linear algebra, vectors are normally thought of as column vectors.
That is, a W and X are both D by one matrices.
The reason I bring this up is because a few students have asked why we don't use W X transpose instead.
Note that this is uncommon because we don't normally treat vectors as rogue vectors and note that it's
also valid to use in product notation or dot notation.
However, you seem to be less common in machine learning.
Now, so far, all we've done in this lecture is described the form of a linear classifier.
What we have not yet done is explain how this turns into logistic regression in order to do that.
We need to introduce the logistic function.
Note that the logistic function is also known as the sigmoid.
In case you've seen different terminology elsewhere, the equation for this function is very simple.
Again, just some tenth grade mathematics in particular, sigma of X is equal to one over one plus the
exponential of minus X.
Note that the equation itself is not too important unless you want to convince yourself of the properties
of the sigmoid I'm about to describe.
Or if you want to implement this yourself in code, so pay more attention to the properties in particular.
Note that this function has two asymptotes on either side as the input approaches infinity.
The output of the sigmoid approach is one on the left side as the input approaches minus infinity.
The output of the sigmoid approaches zero in the middle when its argument is zero.
The function is equal to zero point five.
Importantly, notice that the output of the sigmoid is always between zero and one.
OK, so what is the purpose of introducing this sigmoid function?
The answer is that it allows us to interpret the output of our model as a probability.
In particular, we're going to apply the sigmoid it to our previous expression w transpose x plus b.
We interpret this as the probability that the target Y equals one given X.
As you recall, probabilities must always be between zero and one, which is the case for the Sigma.
Now you might wonder why should we care to output probabilities?
One use case is to compute metrics like the AUC, which can only be done when the model outputs probabilities.
We can even sample from this distribution, which is a common thing to do with neural networks.
One thing to note is that our original decision rule corresponds to rounding the probability we get
back from the sigmoid.
In particular, suppose that our rule is if P of Y equals one, given X is greater than zero point five,
then predicts one.
If P vehicles one given X is less than 0.5, then predict zero.
Otherwise, we fall directly on the line, in which case it doesn't really matter which class you pick.
As long as you are consistent in practice, we normally call the round function.
So if you happen to get exactly zero point five, then we would predict one.
So how does this correspond to our decision ruled from before?
Recall that this is where if a Ovex is greater than zero, then predict one.
If they have X is less than zero, then predict zero.
Otherwise, we fall directly on the line.
This makes perfect sense since this line is the separating boundary.
So if W transpose X plus B is equal to zero, then we are on the line.
But as you recall, the sigmoid of zero is zero point five, which corresponds to a probability of 50
percent 50 percent.
Makes perfect sense because if we fall directly on the line, we are equally sure that this data point
could belong to either class.
Now, using a probability serves another purpose, which is that it allows us to use performance metrics
like the AUC, which can be used when we have imbalanced classes.
As you recall, this is how the RC curve is drawn by tracing at different points for different values
of the threshold.
In other words, we don't always have to use a threshold of 50 percent, even though this corresponds
to the separating boundary.
So, for example, we might use 30 percent if we want to make more positive predictions.
This will give us more true positives, but it will also give us more false positives as well.
Or if your model has too many false positives, you might increase the threshold beyond 50 percent.
That's another way to use the RC is to use it to choose the best threshold based on the tradeoff you
desire between the true positive and false positive rates.
So the next thing I want to discuss in this lecture is to compare logistic regression with naive Bayes
in terms of what kinds of models they are.
One way to categorize them is to say that logistic regression is discriminative while naive bases generative.
The reason for this distinction is as follows logistic regression directly models p of Y given X.
This is what we call discriminative.
It discriminates between the different classes for y directly niveis indirectly models of Y given X,
but it really models p of X given Y.
This is what we call generated.
So why do we call this generative?
Well, a simple reason is that once we have Pivac's given Y, we can actually generate samples from
that distribution.
So given a Class Y sample in X that could have come from this class, this is very easy to do once you
have the distribution p of X given Y.
But you cannot do this if you only have your Y given X.
Now this might seem quite silly in the abstract sense.
Why would you want to generate X?
Well, it helps to think of some examples.
One example is with image recognition.
In this case, X is an image and a Y is the label to recognize.
In this case, if we could sample X given Y, then we could generate random images from a given class.
Gans are one example of state of the art generative models for images.
So, for instance, I could say, given the class cat generate a random image of a cat or given a class
car, generate a random image of a car.