So in this lecture, we will be looking at some neural word embeddings in code.
The first step will be to download some pre-trained word embeddings.
So in practice, we typically use word embeddings that have been trained by someone else.
The reason why we do this is because these methods tend to use very large data sets.
So you don't want to replicate this work if you don't have to.
Now, if you just want to learn, then that's a great exercise, which we do in my other courses that
focus on these specific algorithms.
But for the purpose of this class, that won't be necessary.
In addition, a note that for newer methods like transformers, there is absolutely no way for you to
train those models yourself.
Modern methods can cost millions of dollars per training, which doesn't include all the experimentation
you need to do to optimize hyper parameters and debug and so forth.
So for these methods, you essentially have no choice but to use pre-trained models for word embeddings.
The situation is not so severe, but we typically start with pre-trained vectors anyway.
In any case, the next step is to unzip our zipped file.
The next step is to import key two vectors from Jensen.
This class has the necessary API for interacting with the word embeddings.
We just downloaded.
The next step is to call the function of load words in that format, passing in the pre-trained word
embeddings.
The next step is to implement a function called find analogies.
This function will take, in three words W1, W2 and W3.
Their positions in the analogy are as follows.
We want to have one is to W2 as the result is two three.
So you can see I've represented the result as a question mark in the comments.
For example, we might want to do king minus man equals queen minus woman.
In this case, the question mark should be queen.
If we reorganize the vector so that we isolate the question mark, we get plus king plus woman minus
man.
Basically, the important thing to notice is the sign.
So what we're going to do is call the function most similar.
The inputs to this function are the words which are represented by positive vectors in the words which
are represented by negative vectors.
As you've seen from above, W one and W three should be positive, while W two should be negative.
Note that the result, which have called R, actually contains multiple words along with their corresponding
scores.
Thus, to get the top word only without the score, we index are at zero zero.
The last step is to print out the full analogy so that we can see it in the format we expect.
OK, so the next step is to test our function, starting with our running example.
As you can see, we get king minus men, equals queen minus woman as expected.
Now, let's see if this works for cities and countries.
As you can see, we get France minus Paris equals England minus London.
This makes sense.
Now, let's test this with another city.
So this time we get France minus Paris equals Italy, minus Rome again, a satisfying result.
So you might wonder what would happen if we reverse the order of the country and city, what do we get
if we use country minus city instead of city minus country?
OK, so this time we get a poor result, we get Paris minus France equals Lohan minus Italy.
As far as I know, this does not make sense.
This is our first sign that these vectors are not perfect, although they are pretty good.
Now, let's try.
France is too French as what is too English?
As expected, we get England.
The next step is to try the same kind of analogy, but with different countries.
OK, so this time we get a strange result.
We should expect to get China, but instead we get Tibet.
Of course, these are closely related countries, so it's not completely incorrect.
Furthermore, it's still remarkable that the result is a country and not some random word.
The next step is to try the same type of analogy again, but with Italian instead.
OK, so this time we get Italy as expected.
The next step is to test this with months of the year, so December is to November as what is the June?
OK, so we get September, which is not quite what we expected, but still not bad as it is a month
of the year.
Note that for this type of analogy, I've had better success with glove.
The next step is to test out cities and states in the United States.
So Miami is to Florida as what is to Texas?
OK, so we get Dallas, which is an answer that makes sense.
The next step is to test out occupations.
So Einstein, as a scientist, as what is to painter?
Now for this, I expected to get something like Picasso or Rembrandt.
Personally, I don't know whether or not there is a painter named Jude.
But you can feel free to look that up yourself if you're curious.
At the very least, it is a name which makes sense.
The next thing we're going to test is pronouns.
So, man, is to woman as what is to she.
And the answer is he as expected.
Now, let's try managed to woman as what is two ends.
And the answer is uncle, as expected.
Let's try managed to woman as what is two sister?
The answer is brother, as expected.
Now, let's try man as a woman as what is to wife.
In the answer is Son.
For this, I would have expected to get husband, but again, remember that these vectors are not perfect.
Note that it still may be the case that husband is in the list of returned results is just not the top
result.
Now, let's try managed to woman as actor is to actress.
So the result is actor, as expected.
Now, let's try managed to woman as father is the mother.
And again, we get farther as expected.
Now, let's try nephew is the niece as uncle is two aunt.
And again, we get uncle as expected.
OK, so recall that what we've done is mapped words to vectors, we've so far looked at analogies,
but an even simpler question is to ask, given a word, what are the most similar words to do this?
We're going to write a function called the nearest neighbors that takes in only one word.
In fact, this function is much simpler than before because we only have to specify the positive arguments
once we get the result was simply going to loop through them and print out the matching words.
OK, so let's look at the nearest neighbors for King.
So as you can see, we get Kings, Queen, Monarch, Crown Prince, Prince Sultan and so forth.
So these are seem to make sense.
Now, let's try France.
So we get Spain, French, Germany, Europe, Italy and so forth.
This all makes sense since they are countries except French, which is not a country, but still makes
sense in Europe, which is a continent.
But notice that all these countries are in Europe.
Now, let's try Einstein.
So in this case, we get some pretty weird results.
Most of these are names, which makes sense.
Albert is somehow in third place, which is odd, and somehow Ellen Fellow is in second place.
Perhaps this has to do with the fact that people often use the term Einstein sarcastically.
Now, let's look at the nearest neighbors for a woman.
In this case, we get man, girl, teenage girl, teenager and so on.
So these mostly make sense.
Now, let's try, nephew.
So this time we get Sun, uncle, brother, grandson, a cousin and so forth.
So these are make sense.
Now, let's try February, which is a month of the year.
OK, so as you can see, most of the other matches are months, which makes sense.
So now that you know how word vectors work, it's time to do an exercise.
We've just looked at how to analyze words Buvac vectors.
This was relatively easy because we had the Genzyme Library to do most of the work.
However, note that this work is not really that difficult.
Once you have the vectors for this exercise, you're going to download glove vectors, which gives you
the actual vector is stored as text instead of a binary file.
Like Google's words of that, this file is a very simple format, which you should be able to figure
out.
Once you've downloaded the glove vectors, you should write the code to implement, find analogies and
nearest neighbors to work exactly the same way as they did in this notebook.
Your goal is to check whether or not glove vectors are better than words of X..
Good luck, and I'll see you in the next lecture.