OK, so in this lecture, we are going to discuss the concept of vector similarity.
Basically, you can think of this as a function that will take in a two vectors as input and as output.
We will get back a single number, which tells us how similar those two vectors were.
You can think of this like a score.
So why is this useful?
Well, suppose that we were successful in mapping all of the documents in our text corpus into vectors,
perhaps using TFI Taf.
Now we can ask questions like given some document, find me another document in the corpus that is most
like the given document.
You can imagine this might be useful for researchers who are looking for journal papers on a specific
subject.
We can also ask the opposite question which document is most unlike the given document?
We can also do the same thing with words.
Suppose that we've managed to map words into vectors.
We can then compute the similarity between words.
So, for example, one might expect the words king and queen to be very similar.
Car vehicle and automobile should also be considered similar.
Conversely, car and drive should be very dissimilar.
This could be useful, for example, if you wanted to build an article spinner where your goal is to
rewrite an existing article, but to replace every few words such that your new article is sufficiently
different from the existing one.
This technique is used by marketers who want to build websites with lots of content.
But, of course, don't have the time or knowledge to write that content themselves, and also may not
have the funds to pay others to write the content for them.
Article spinning is a way to automatically generate content that won't be penalized by search engines
like Google for being too similar to existing content.
Now, I personally would not advocate doing this, since it's considered a Black Hat SEO technique,
but it's an example of how machine learning could be used.
Ultimately, whether it is ethical or not, it is one method that marketers have used to make money
in the past.
So word replacement is one application of vector similarity.
OK, so as you recall, when we talk about vectors in machine learning, we typically assume they live
in a Euclidean space.
So each component is the length of the vector in that respective direction.
As a simple example, the vector of three four means go three units in the X One Direction and go for
units in the X two direction, and that will give you the tip of the vector.
If you think of it as an arrow or simply the data point representing that vector.
Now, because of this, there is a very natural way to define a similarity of vectors, and that is
in terms of distance.
Essentially, distance is the opposite of similarity when two vectors are close together in distance.
They are similar when two vectors are far away in distance.
They are dissimilar.
In this case, we can think of similarity as simply the negative of distance.
So how do we define distance and Euclidean space?
Well, one typical way to define distance is to use the Euclidean distance.
Of course, this makes complete sense Euclidean space, Euclidean distance.
So let's review how to compute the Euclidean distance.
So if we have two vectors X and Y each with size D, then what we do is we take the component wise difference
of each corresponding component.
Square those differences, add them together and then take the square root of the results.
Obviously, if you want the squared Euclidean distance, then simply do not take the square root.
And the reason we can use the square of the Euclidean distance is because the square is a monotonically
increasing function.
Thus, if Point A and B have a smaller Euclidean distance compared to you point A and point C, then
it is also true that point and point B have a smaller squared Euclidean distance compared to point A
and point C.
In other words, when the only thing you care about doing is comparing relative distances, then it
doesn't matter whether you use the squared Euclidean distance or just the Euclidean distance.
Now, Euclidean distance is not the only way to define vector similarity.
Another way to define a similarity is by looking at the angle between two vectors, specifically the
cosine of the angle.
So suppose we have two vectors X and Y.
Recall that the inner product can be defined in two different ways.
The first way to define the inner product is that it's the sum of the Element Y's product of the two
vectors.
So that's x one times y one plus x two times y two and so forth.
Now, the second way to define the inner product is this it's the magnitude of x times the magnitude
of Y times the cosine of the angle between X and Y.
Of course, the cosine of the angle is what we want to solve for.
And so if we equate the two forms of the inner product, this is what we get.
It's the sum of the Element Y's product of the two vectors divided by the magnitudes of the two vectors.
So what is the cosine similarity?
It's simply the cosine of the angle between two vectors, but we just looked at how to compute.
So let's think about why this works.
Firstly, it helps to recall what the cosine function looks like.
So it's a periodic wave, but we are only interested in the values from zero to PI.
Now, suppose that we have two vectors which are very close to being parallel.
The angle between them is about zero.
In this case, the cosine of this angle is about one, which is the maximum value for the cosine.
That is to say, this is when the two vectors are most similar.
Now, suppose that we have two vectors which are A. parallel that is facing opposite directions in this
case, the angle is 180 degrees and the cosine of this angle is minus one.
This is the minimum value of the cosine.
And this corresponds to when the two vectors are at least similar.
OK, so now we're convinced that using the cosine of the angle between two vectors works as a measure
of similarity.
Note that sometimes people refer to the cosine distance, which is one minus the cosine similarity.
So how does this work?
Well, again, we can see that if two vectors are parallel, then they are as similar as possible.
In this case, the cosine similarity will be one and one minus one is zero.
So the cosine distance is zero and this is the minimum possible cosine distance when there is zero angle
between the two vectors.
On the other hand, if two vectors are A. parallel, then the cosine similarity will be minus one and
one minus minus one is two, which is the maximum possible cosine distance.
So whether you want to work with cosine similarity or cosine distance, the principle is exactly the
same.
Note that, strictly speaking, the cosine distance is not a true distance.
This is because it doesn't satisfy the axioms of distance metrics, specifically the triangle inequality
at the same time.
This doesn't really prevent us from doing anything we want to do.
So it turns out that cosine similarity is a bit more common than Euclidean distance in NLP.
If we think about vectors form from counting, we can think about why this would be suppose for simplicity
that we have two words we are counting mitochondria and voltage.
So intuitively, we know that mitochondria would appear more often in books about biology and the life
sciences, while voltage would appear more often in books about electronics or physics.
Now, suppose that we have one very long book where mitochondria appears 500 times and we have one very
short book where mitochondria appear as just a few times.
Now, suppose we have an electronics book where voltage appears just a few times.
What happens when we look at the Euclidean distance between these vectors?
Well, something surprising happens.
It turns out that the electronic book is closer to our short biology book than the short biology book
is to the long biology book.
And this doesn't make sense simply by virtue of having fewer words overall.
We can see that two completely unrelated books have a smaller distance compared to two books on the
same topic.
This might suggest that perhaps Euclidean distance is not the best choice.
Based on this example, the cosine distance would be zero between our two biology books.
Well, it would be one for the biology book compared to the electronic book.
Thus, when we have to deal with it, vectors of different sizes cosine distances might be more useful.
So there is one scenario where a cosine distance and Euclidean distance yields equivalent results.
Suppose that we don't care about distance or similarity value, but instead just the relative distance
or similarity between different items.
For example, if you're building a recommendation system or a search engine, you don't care about the
score itself.
You only care about the ranking of the items after you've sorted them by the score.
So let's suppose that you've normalized your feature vectors now that you know their magnitude can have
an effect on the outcome.
That is, the L2 norm of each vector is one.
In this case, we can note the following.
We'll start with the squared Euclidean distance between X and Y.
Recall that this is simply equal to the vector x minus Y dotted with itself.
The next step is to expand this expression, which gives us x transpose times x minus two times x transpose
times y plus y transpose times y.
However, we've just established that X transpose times X and Y transpose times y are both one since
we normalized X and Y.
So this gives us two minus two times X transpose y.
Furthermore, as you recall, it's transpose Y is just the magnitude of X times the magnitude of Y times
the cosine of the angle between X and Y.
But the magnitudes of X and Y again are just one.
Therefore, we end up with two minus two times the cosine similarity.
Or put another way two times the cosine distance.
And of course, multiplying by two has no effect on the ranking of items which use this as a score.
What this means is that if you normalize your vectors and the only task you need to do is rank items,
then it doesn't matter which similarity or distance you use.