So in this lecture, we will briefly introduce the concept of neural word embeddings.
Now this topic is a bit more advanced than the rest of this section, so we won't go into great detail.
However, I would like to introduce this topic now so that you have at least some idea of the other
ways we can create vectors out of text.
When you go on to study deep learning for NLP, then you will be able to study these methods more in
depth.
But for now, this is just a brief introduction about what is possible.
So in the previous lectures, we saw two methods of converting documents into vectors.
That is, we converted each document in a machine learning dataset into a vector where each component
of that vector corresponds to a different unique word neural embeddings or a bit different.
Firstly, note that the word embedding is just a fancy way of saying vector, at least for our purposes,
so don't be intimidated by different terminology.
Secondly, these neural methods are typically used to convert words into vectors instead of full documents.
Thus, we get a much more fine grained representation.
If each word can become a vector than a document, then becomes a whole sequence of vectors.
This contains much more information than what we looked at in this section, where the whole document
gets turned into just a single vector.
As you recall, this is called a bag of words.
So why might sequences of vectors be useful?
Well, in machine learning and specifically deep learning, we have special models that are purpose
built for sequences.
These include CNN's ANA Ends and Transformers.
Unlike bag of words methods, these models do consider the ordering of the words in a sentence.
These methods are so powerful they have led to state of the art performance in many areas.
For example, language translation question answering chat bots, speech to text, text to speech and
biological sequence analysis.
Now, there are two methods I like to discuss when it comes to word embeddings.
These are where to effect and glove.
At this point, there's no need to go into any detail since we haven't discussed the necessary prerequisites.
However, I can share with you the kinds of techniques which are used.
So four words of EC, what you need to know is how to build a feedforward neural network in order to
find the word embeddings.
We need to find the weights of the neural network, which effectively are the word embeddings themselves.
The goal of training this neural network is quite simple.
Although the details are more complex.
Basically, the goal of this neural network is to predict, given an input word, whether or not another
word would be found in its context by context, we mean a small window surrounding the input word.
So consider the sentence the quick brown fox jumps over the lazy dog.
Suppose that our input to the neural network is jumps, and the size of the context window is five words,
which is two to the left and two to the right.
So the output brown would be considered a positive match, as would Fox over in the, on the other hand,
dog would be a negative match.
Furthermore, any word not found in the sentence, such as dinosaur, would also be considered a negative
match.
So that's words of EQ.
You're essentially training a neural network to make a binary prediction about whether an input word
and some output word can be found close together in the text corpus.
So Glove doesn't use neural networks, but it was invented at the same time that deep learning became
popular for NLP.
The word embeddings that come from glove are also used in neural networks at later stages.
And thus, although we don't technically use neural networks to find the glove embeddings, it is still
considered part of deep LP.
So how does glove work?
Glove essentially works like a recommender system, suppose that you have a list of 1000 users and 100
movies.
Not every user has watched every movie and thus there are still items to recommend.
Your goal in training a recommender system is to take the existing ratings and to try to predict what
users will rate movies they have not yet seen.
The idea being that if you predict a user will rate a movie highly, then you should recommend that
movie to that user.
So how does this relate back to love?
Well, just like we're Divac, we apply the idea of context.
Imagine again this sentence.
The quick brown fox jumps over the lazy dog.
Suppose again, we take the word jumps.
We can see that jumps is just one word away from Fox, so perhaps we may give that a score of one out
of one.
We see that jumps is two words away from brown.
So perhaps we may give that a score of one out of two.
So you can see that we are pretending that in this recommender system, the users are words, but the
movies are also words.
Our so-called ratings are simply based on the distance between the words.
And that's because of this analogy.
We can build a recommender system out of words.
How high one word would rate another word.
So that's the essence of glove.
Now, please keep in mind that I do have material for you if you want to learn the details about how
words affect and glove work.
So if you're eager to learn about them, please look through the rest of the course materials or ask
me on the Q&A if you can't find what you need.
OK, so now let's get back to the practical world.
What can we actually do with word embeddings?
One simplistic way to use the word embeddings is to convert a full document into a vector, just like
we did in this section.
However, these vectors, unlike count vectors and TFI Taf, will not be sparse and will not be high
dimensional.
This is because we typically choose a small number for the vector, a dimensionality such as 20, 50,
100, 300 and so forth.
So these vectors are very small, unlike the vectors from this section, which have size V.
So how does it work?
Well, suppose we did the following steps given a document, we first tokenize it into individual words.
The next step is to use our word embeddings to map each of those words to their corresponding word vectors.
Once we have those word vectors, we simply take their average to get a single vector of the same size.
This gives us one vector that represents the whole document without regard to the ordering of the words.
Thus, it is another bag of words representation.
Another interesting thing you can do with word embeddings is word analogies.
This is perhaps the most impressive result of these word embedding techniques because what we have is
a set of vectors we can do arithmetic on these vectors, that is, we can add and subtract.
So consider the analogy king is to man as what is to woman.
The answer, of course, is queen.
This is because kings are men and queens are women, and they both occupy sort of the same role in their
society.
It turns out that we can express this analogy in math.
We can say that the vector for king minus the vector for men is approximately equal to the vector for
queen, minus the vector for a woman.
The way we actually do it in code is this.
Suppose we want to answer the question, as I asked it.
King is to man as what is to woman.
In this case, we would take king minus man, a plus woman.
This would give us some arbitrary vector.
Then we would search through our list of vectors to find the closest one to this vector.
The result is that we find queen as expected.
So it turns out that we can do this type of analogy for all sorts of things.
For example, France is to Paris, as Italy is to Rome.
So that's with countries and cities.
We can also do Japan as the Japanese, as China is to Chinese.
So that's the country and what we call the people of that country.
We can also do Miami is to Florida, as Dallas is to Texas.
Thus, the model knows about how cities are related to states.
We can also do December is to November as July and June.
Does the model know something about the ordering of months?
We can also do manage to woman as he is to she.
So the model knows something about pronouns.
Now at first glance, these might seem like just toy applications.
You might ask How is this going to help me predict spam or predict sentiment?
The answer is that these are just small parts of a bigger machine.
That machine being neural networks, what this tells us is that we are able to learn to some sense of
meaning for each of these words by their position in a vector space that by itself is pretty profound.
And what this tells us is that if we were to use these vectors in a neural network, then at least we
know that this part of the neural network has close to the optimal weights, and thus we could perhaps
avoid having to train those weights.
You can think of this like transfer learning, where you transfer knowledge from one task to another
task.