So in this lecture, we will be discussing embeddings, which are the deep learning way of converting
words into vectors to begin this lecture.
Let's step outside of NLP for a little bit.
Suppose we're just doing some generic machine learning task, like trying to predict whether or not
a user will make a purchase on your website.
One of the data points you have is how the user got to your website.
There are three possibilities.
Either they got there organically, like searching on DuckDuckGo, or they got there from an advertisement,
or they got there from an affiliate.
This data is categorical, so we cannot just put them into our end by the matrix like we normally would.
So how can we include this data in our machine learning model?
One idea you might have is to simply assign a numbers to these possibilities.
So organic becomes zero.
Advertisement becomes one.
An affiliate becomes, too.
This is not a good approach, since this puts these different categories on a scale.
Numerical data only makes sense when your data actually is numerical.
For example, like using the number of hours you studied for an exam to predict your exam grade, but
there's no reason for these particular numbers to be assigned.
For example, why can't we make organic minus one million instead?
As you may recall, one popular option is a one hot encoding using this method.
This particular input becomes a vector of size three one.
Hot encoding means that for each of these possibilities, they will get a unique vector representation
where one of the positions in this vector has the value one, while the rest are all zeros.
So, for instance, organic becomes one zero zero.
Advertisement becomes zero one zero.
And affiliate becomes zero zero one.
Now, this makes sense.
Imagine if you build a model using logistic regression, as we've done in the past.
Suppose that the wait for the first position is very large and positive.
This means that users who came to your site organically are more likely to make a purchase.
Suppose that the wait in the second position is very negative.
This means that users who came to your site from an advertisement are less likely to make a purchase.
Now, clearly, this is directly applicable to an LP since no deals with words and currently what we
are trying to do is to figure out how to convert text into a numerical feature vector, which is appropriate
for a neural network.
If we treat each word as a category, then all we have to do is create a very big feature vector with
one position for every possible word.
Once we have this vector, which is a vector of mostly zeros except for a single one, we can then use
the usual W Transpose X plus B computation, which you've seen many times.
Unfortunately, there's a problem with this because this vector is very big.
This matrix multiplication will be very slow.
So consider what happens if we multiply a one hot encoded vector with a weight matrix.
Let's suppose the vector is just three dimensional one zero zero.
And the way Matrix just contains the numbers one up tonight.
You can verify for yourself that when we multiply this vector by this matrix, we get the vector one
two three.
Now, consider the one hot quoted Vector zero one zero.
What happens when you multiply this vector by the same way, matrix, we get the vector of four or five
six.
Now, consider the one hot and quoted Vector zero zero one.
What happens when we multiply this vector by the same weight matrix?
We get the vector seven, eight nine.
So what is the pattern we see if the index in the one hot and coded vector, which was set to one was
one.
Then we get the first row of the weight matrix if the index was set to two.
Then we get the second row of the weight matrix if the index was set to three.
Then we get the third row of the weight matrix.
In other words, if we one hot and cold at the integer K and multiply it by the weight matrix, all
that's really doing is selecting the fifth row of the weight matrix.
So here's a shortcut.
The old way of doing this took two steps.
First, we had to create a one hot and coated vector and set the key for entry to one second.
We had to multiply this one hot and encoded vector, by the way matrix.
The shortcut way of doing this takes only one step.
We simply index the weight matrix at the rock.
This is obviously much more efficient than creating a one shot vector and then doing matrix multiplication.
Think of it this way.
Indexing an array is of one that's constant time.
But how long does it take to create a one hot vector and then do matrix multiplication?
If your vector is a size V and the weight matrix is of size v times D, then this is o of v times D.
In fact, what I've just described is the embedding layer, the embedding layer is just like a dense
layer.
If you passed in a one hot and coated vector and you didn't have any bites terms because of what we
just learned.
We know that if our input is a one shot encoded vector, then it's much more efficient to simply select
the appropriate role of our weight matrix instead of doing a full matrix multiply.
So let's consider how we will use the embedding layer in code.
As usual, we start with an input layer.
For this input layer, I've specified that the input dimension is just T.
What this means is that we'll be passing in sequences of lengthy.
The next step is to pass that input into an embedding layer, as you've seen internally.
This is just a weight matrix, and thus we have to specify both the number of inputs and the number
of outputs.
The number of inputs should be our vocabulary size.
This is because we should have a row for each unique word.
The number of outputs should be the vector dimensionality that we choose for our word embeddings.
This is another example of a hyper parameter to be optimized by the user.
Now, as you know, it's always important to think about shapes.
Suppose we called model that summer and we looked at the shape of the output after our embedding.
What we would see is that our data now has three dimensions and by T, by D, or, in other words,
number of samples, by sequence length, by embedding dimension.
Of course, this makes sense.
If we passed in any documents, we should still have any samples.
If all of those documents contain a T words, then they should still have the sequence length T.
The key is that each of those words was converted into a d dimensional vector and therefore our output
has the shape end by T by D.
Understanding the shape is critical for understanding CNN's inheritance.
Essentially, you can think of this like a multidimensional time series data sets after which you could
pass this into a CNN or Arnon like you normally would.
At this point, I would like to take a moment to step back and to think about the big picture of what
we have really done.
Originally, I said we should a one hot encode each word, which technically is still the case.
We just haven't explicitly created the one hot and coatings because that's inefficient.
But notice how such one hot and coded vectors are not useful geometrically.
As you recall, one of my mottos for machine learning is that machine learning is nothing but geometry.
When we plot word vectors, they should exist in some useful vector space.
So, for instance, if I searched for the nearest neighbors of cats, I might find words like feline
lion, cougar and so forth.
But notice how one hot and coded vectors do not have this characteristic.
If you take any two one hot and coded vectors, the distance between them is always the square root
of two.
So Cat is as close to feline as it is to an unrelated word, such as airplane.
Effectively, what we are really doing with an embedding is creating a table of word vectors.
The embedding layer is like a database for these word vectors.
The word index is like a query into that database and the value or the output is the word vector corresponding
to that word.
It is our hope that these vectors will have some useful structure.
So if we search for the nearest neighbors of cat, we will find words like feline.
From this perspective, we have not exactly one hot encoded each word and multiply that by a matrix.
But instead, we've simply come up with a better vector representation for each word in the next lecture.
We'll look at one method to find such vectors.