So in this video, we are going to discuss support vector machines.
Now, luckily, thanks to the previous two lectures, you now understand the basics of linear models.
This lecture is kind of a different way of putting a linear model together.
So the easiest way to start this discussion of support vector machines is with classification.
As you've heard me say before, this is nothing but a geometry problem.
Now we know how logistic regression finds this line.
It does so by minimizing a lost function, which happens to be the negative log likelihood of a coin
toss distribution.
The details aren't too important, but it's a model based on probability.
In contrast, the support vector machine is a model based completely on geometry.
So how is the support vector machine based on geometry?
Well, the SVM is what we call a maximum margin classifier using this data set as an example, we can
see that there is definitely more than one line that can separate the two classes.
But how do we define what a good line is?
Intuitively, you might say that this line is not a good line because it doesn't generalize.
Well, it's too close to the data points.
If I add a new data point here, it's pretty obvious which class this belongs to, and yet this line
would classify it as the wrong class.
The intuition behind the SVM is that the best line is the line that is as far away from the points as
possible.
You can see that according to this criterion, this line is the best line, because if we rotate it
in any direction, it becomes a worse line going closer to other data points.
The space between the closest data points is called the margin, hence the SVM is a maximum margin classifier.
It tries to maximize this margin.
As mentioned, you can see that this is based completely on geometry.
Furthermore, notice that there are data points which fall on the lines which define the boundaries
of the margin.
These are input vectors which are from our data set because they fall directly on these lines.
We call them support vectors.
Hence why this model is called the support vector machine.
These support vectors are in some sense supporting this line with the corresponding margin.
OK, so hopefully that makes sense why this is called a support vector machine.
So basically the methods of solving this geometry problem are well-established.
So bringing this from concept to code is technically not too difficult.
If you're a person who enjoys working on geometrical problems, I'd encourage you to think about how
this might be solved.
Now, the picture we just discussed is pretty nice, but to ideal, we know that in the real world,
data is not completely separable.
That's why when you train a model, you get 95 percent accuracy and not 100 percent accuracy.
In fact, the methods used to solve the geometrical problem will fail in the case where your data points
cannot be perfectly separated by a line or a plane.
So the solution to this problem is to introduce variables like variables are basically a way for the
optimization to cheat.
Instead of trying to only maximize the margin, we allow a few data points to not be classified correctly.
When this happens, it simply increases our objective function.
So in effect, our optimization ends up being a balance between two terms.
On one hand, we'd like to maximize the margin.
On the other hand, we'd like to have the minimum possible amount of slack.
Of course, this balance can be controlled by the user and it would depend on the problem at hand.
OK, so those are the basics for the support vector machine when we want to do classification.
Remember that this was only intuition.
There are still details we haven't discussed, like how one would handle multiple classes.
You're encouraged to check out extra reading text if you'd like to go into further detail.
The next topic for this video is how do EVMs do a regression?
That is, instead of predicting a category, how do we get the model to predict a number?
So the regression case is kind of interesting because it's not at all clear how this is related to the
model we just built for classification.
There's no real concept of a margin that we want to maximize.
Unfortunately, the link is not really easy to understand unless you've seen the math.
So consider learning this topic in depth if you're curious.
Now, the bright side to this is that understanding the intuition behind SVM for regression is just
as easy as it was before using a picture.
So what is this picture saying?
Well, on the left, we have a picture of our line of best fit, and on the right we have a picture
of the corresponding loss function.
This loss function is called the Epsilon insensitive loss.
The basic idea is there any point within Epsilon distance of the line does not contribute to the loss.
In other words, its losses zero.
In other words, if the line is close enough to all the points that is considered acceptable only when
points go outside the Epsilon boundary do they have non-zero slack.
In this case, the loss would increase above zero and therefore the optimization problem is to minimize
the slack and make it so that as many points as possible are within epsilon distance to the line.
Contrast this with the squared loss, which is positive for every point, which isn't precisely on the
line.
Now, one question worth asking at this point is why should we do all this work just to get yet another
linear model for regression and classification?
In fact, this is not the true power behind this film.
What makes this film really powerful is the fact that it employs something called the kernel trick in
order to find non-linear models.
And again, in order to understand this, you really have to understand the math.
So just accept this as fact until you do.
The simplest way to understand the SVM is by understanding feature engineering.
As you may have learned in the past, one simple way to create nonlinear models out of linear ones is
to simply create non-linear features, for example, X squared, X cubed and so forth.
In general, we can think of a generic feature transformation called FI, and by applying this before
we use our linear model, we effectively get a nonlinear model.
The most popular kind of feature engineering for the SVM is called the RBF Kernel.
So what is the RBF, Colonel?
Well, again, it helps to look at a picture, as you recall, our data comes in the form of vectors.
So imagine some vector space.
Now, imagine that we have some feature vector X and we would like to feature it.
As you recall, what is really important in SVM theory is the support vector.
Sometimes we call these landmarks or exemplars.
Basically, they are the points that really matter.
And so what we effectively want to know is how close is X to each of these landmarks?
Suppose that we have two landmarks, or in other words, to support vectors called L1 and L2.
We want some measure of similarity to tell us how close X is to L1 and L2.
Since X is close to our one, we want that to yield a larger number like zero point nine since X is
far from L2.
We want that to yield a smaller number like zero point zero one.
And note that this measure of similarity always gives us the number between zero and one.
So the army of colonel or the similarity function is essentially based on the Gaussian.
It's very easy to see how this works by plugging in the extreme cases.
If X is equal to the landmark and the distance between them is zero, the exponent of zero is one.
And so the similarities maximum.
If X is infinitely far away from the landmark, then the distance between them is infinity.
The exponent of minus infinity is zero.
And so the similarity is minimum.
Not that we can control how sensitive this function is by using this hyper parameter called beta.
Based on this value, you can control how skinny or fat the bell curve is.
Now, you might be wondering, why is the army of colonels so special?
Well, recall that earlier I said something about applying some feature transformation phi to your input
to obtain a non-linear model.
But FI is not the kernel.
The real power behind the SVM is that it allows you to use Col's instead of computing PHY.
This is helpful, especially when FIA's a very complicated function, perhaps even impossible to compute.
And it turns out that after doing a lot of math, you can prove that the RB of COL corresponds to an
infinite dimensional feature expansion.
In other words, it certainly is impossible to compute directly.
So using the kernel is a trick that allows you to compute something that would be otherwise in computable
simply by calculating the distance between your input and the support vectors.
Again, it's hard to appreciate this without going through the math, but I hope that this gives you
at least some idea of what's going on.