In this lecture, we are going to discuss a tiny bit of theory related to linear classification.
Previously, we discussed that linear regression.
As you recall, this is the task where our goal is to find a line or curve of best fit.
You want your model to output predictions, which are close to the data points in classification.
This is not our task in the classification, but we do not want the line to be close to the data points.
Rather, in classification, our goal is to separate data points into distinct classes.
Although we can consider the case where we have more than two classes, we'll start with two since that
is the most intuitive.
Because this is linear classification analogous to linear regression, our model is still a line.
It's just that the job that we want this line to do is different from regression.
Unlike regression, we don't care if this line is close to the data points.
Instead, we want the line to separate data points of different classes.
In other words, if one class consists of the red dots and one class consists of the blue dots, then
we want all the red dots to be on one side of the line.
And we want all the blue dots to be on the other side of the line.
In this way, we can say that the line discriminates between the two classes.
As you can see, this is not related at all to having the line be close to the data points.
Also, I hope I haven't lost you in terms of what the colors of the dots mean.
Remember the rule?
All data is the same.
So for one particular data set, the color might represent whether a credit card transaction is fraudulent
or not.
For another data set, the color might represent whether an email is spam or not.
For another data set, the color might represent whether a student will graduate or not.
And as you can see in this example, it represents disease versus healthy.
As with the previous lectures, we are going to start from a very basic perspective.
How would this work inside get learn?
We're going to learn what each of these steps mean in terms of concepts, and we'll also learn how each
of these steps are done in PyTorch.
Remember that in PyTorch there is no model constructor or fit function or predictor function, so we
have to do all that ourselves.
To recap.
Here are the steps we do in a typical machine learning script with socket learn.
First we are going to load in some data we call that X and Y.
Next, we're going to instantiate a model.
Next we are going to train or fit the model using models that fit x y.
Next, we are going to make predictions with the model using model that predict X.
This can be the same X or a different x, for example, x train or x test.
Finally, we can evaluate our model using model that score x y.
For classification, the score returns the classification accuracy.
And just to reiterate, this lecture is all about answering the question what actually happens inside
these functions?
This question is really important to answer because you'll notice that if you compare this to linear
regression, literally nothing has changed except for the name of the class.
So really what happens inside these functions makes all the difference.
It's what really matters if you want to know how this stuff actually works.
As a side note, if you're a beginner and you don't yet know what classification accuracy means, here's
a quick definition.
As you know, our model is a binary classifier.
It's predicting spam or not spam, for example.
Therefore its predictions can only be right or wrong.
There are no other possibilities.
Either it's right or it's wrong.
The classification accuracy, then, is simply the number of predictions I get right.
Divided by the number of total predictions.
The classification error is the number of predictions I get wrong.
Divided by the total number of predictions.
It should be easy to verify that the classification error is equal to one minus the classification accuracy.
Also note that you can calculate the accuracy and error on both the train antacids so you can speak
of the train accuracy and the test accuracy which would be evaluated on the train set and the test set
respectively.
So the first concept we want to consider is what is the model?
More accurately, we want to ask, what is the model architecture?
As usual, it helps to look at a picture.
So here is a typical picture of a linear classification problem.
We have some data points and we want to separate the data points of different colors with a line as
a sign.
So why do we call this linear classification?
Well, you will notice that the word line actually appears in the word linear.
This surprises many people.
So don't worry if you haven't noticed this before.
All right.
So what's the equation for a line?
I claim that a line can be expressed using the equation w1x1 plus w2x2 plus b equals zero.
Now at this point, you might say to yourself and say, Gosh, why does lazy programmer have to make
everything so complicated?
I already know that the equation for a line is Y equals m, x plus B, so why do we have to use this
more complicated looking equation with WS and x one and x two and so on?
In fact, we must.
You'll see why this form of the line is useful very shortly.
If this is new to you, I would recommend the following exercise.
First, notice that all we've done is rename the axes from X and Y to x, one and two.
This is important because these axes refer to the input data.
If you recall, y is the target.
For example, if we're going to classify images of cats versus images of dogs, then we might say dogs
are the red dots and cats are the blue dogs, but they are not an axis on this graph.
So x one represents the horizontal axis and x two represents the vertical axis.
Y is not an axis.
This is different from linear regression where the target y was an axis on the graph.
Now that you know this, you should be able to rearrange the equation w1x1 plus w2x2 plus b equals zero
and two slope intercept format.
What I mean by that is have x two on the left side by itself and then you'll have some slope times x
one plus some intercepts.
Now it has the same format as y equals m x plus B, except that we use x one and x two instead of x
and y.
Also note that the B that appears in Y equals m x plus b is not the same as the B that appears in the
other equation.
So in this slide, I've called the intercept B prime to differentiate it from the original b.
As always, if you can't immediately see how I got this, you should try it by yourself on paper so
you know how to do it.
It should only require elementary school algebra.
The next question you might have is how does this line help us make predictions?
Luckily due to the rules of geometry, if we plug in a data point X which is not on the line, then
either we will get a number bigger than zero or less than zero.
In fact, any data point on one side of the line will always give us a number bigger than zero.
Any data point on the other side of the line will always give us a number less than zero.
Using this, it's very easy to turn this into a prediction model.
All we have to do is take in any data point x, which is a vector containing the elements x, one and
next to pass it into the expression for our line w one, x one plus w two, x two plus b and then check
it sine.
If the sign is positive, we predict one.
If it's negative, we predict zero.
Mathematically, you can encapsulate this decision rule using the step function, or if you want to
be less formal, you can think of it as the picture that you see here.
We call the activation.
And if the activation is greater than zero, we predict one.
Otherwise we predict zero.
Now, as you may already know, in deep learning, we really like differentiable smooth functions.
So rather than the step function, which is not smooth, what we do is take a smooth version of this
called the Sigmoid.
The Sigmoid is an S-shaped curve, and it maps the activation to a number between a zero and one.
We usually interpret this as the probability that Y equals one given X.
Then when we want to make our prediction, we just round this probability.
So if the probability is greater than 50%, we predict one.
Otherwise, we predict zero.
Again, this is called the sigmoid function, which is important to know for our implementation.
As a side note, when we apply the sigmoid function on top of a linear function, we call this model
logistic regression.
This is because we also sometimes call the sigmoid function in the logistic function, although this
term is not really used too often these days.
In addition, we also refer to the argument into the logistic function as the largest but a more current
and generic term for this is activation.
You might realize that using what we know so far, there is a little bit of a notational challenge.
If we keep writing each component of X separately, we're going to run out of space.
It's easy to write w1x1 plus w2x2 when there are only two components of x.
But what if there are 100 or 1000?
Luckily, we have a mathematical way of representing this.
If we consider X to be a feature vector containing each component of X and W to be a weight vector containing
each component of W, then our expression just becomes the dot product between one x.
So we can write this in a much more compact way by saying probability of y equals one given x is equal
to the sigmoid of W transpose x plus b.
Okay.
So at this point, let's recap the three concepts we need to cover.
Number one, model architecture.
Number two, using the model to make predictions.
And number three, model training we just covered.
Number one.
And number two.
You know what the model architecture is and you know how to use it to make predictions.
Basically, given some input X, you know how to plug it into the previous formula to get some output
prediction y hat.
The final step is training.
As I promised earlier, we are going to follow the exact same general steps as we did when we learned
about linear regression.
It may surprise you, but these are the same steps that we are going to follow for every subsequent
example in this course.
That's why I said it's the simplest example.
But it was also the most important.
Now you see why.
So as you recall, we need to define a loss function.
Then after we've done that, all we need to do is apply the gradient descent procedure to update the
parameters of the model in order to minimize that loss.
Once we've completed training, we can plot the loss per iteration to ensure that the loss converts
successfully, and we can test our model by checking its accuracy on the train and test sets.
The main difference in the training process between regression and classification is that we have a
different loss function.
As you recall, in regression our target is a real number and our prediction is also a real number.
In this scenario, it makes sense to use a loss like the mean squared error or MSI.
But in classification our target is a category.
Our model output is the probability that the input belongs to each of the categories.
So the mean squared error doesn't make any sense in this scenario.
In fact, for classification, what we want is the cross entropy loss.
There's one little wrinkle here, which is that we're doing binary classification in which there are
only two possible classes, and the corresponding loss is called the binary cross entropy loss.
In general, if your model can handle any number of classes, then you'll use the regular cross entropy
loss.
But we'll discuss that later in this course.
The reasoning behind the cross entropy loss is quite a bit more complicated than the mean squared error,
so we won't discuss that theory in this lecture.
If you want to learn more, then you should check out the in-depth sections of this course.
For now, all you really need to know is the PyTorch API.
Since as you recall, this course is not about mathematical theory but rather how we do things in pytorch.
Therefore your job really is knowing the right function to call and spelling it correctly.
Indeed, in this computer science course, what matters isn't your math skill but your spelling ability.
To summarize this lecture, we went over the concepts behind linear classification.
Our goal was to consider these three things.
Number one, what is the model architecture?
Number two, how do we make predictions?
And number three, how do we train the model?
As you saw, it was very similar to linear regression in the sense that our model is still a line or
a hyper plane, but it's how we use this line that's different in regression.
Our goal is to get the line close to the data points, but in classification, our goal is to separate
the data points.
In regression, we just pass our data into W times X plus B and that is our prediction.
But in classification we still pass our data through W, times X plus B, we just have the additional
steps of applying the sigmoid function and rounding the upward probability for number three.
Again, it was very similar to our previous example where we set up a loss function and then use gradient
descent to find the weights that minimize it.
But the difference in classification is that instead of using the mean squared error, we use the cross
entropy.
And in our special case of binary classification, we use the binary cross entropy.