In this section of the course we are going to discuss how to deal with tax data in natural language
processing.
This directly ties into the previous section which was on using R and ends for modeling sequence data.
Of course text is also sequence data but there is a major difference between tax sequences and the continuous
valued sequences we were dealing with in the previous section.
That differences.
Text is made of words and words are categorical objects.
In other words they are not continuous
suppose we're given a sequence of words.
The quick brown fox jumps over the lazy dog so x1 equals the X two equals quick and so on.
But here's the problem we know that in order to calculate the output of an hour and end unit we have
to multiply each x by the inputs ahead and wait w x H.
You can see that if X is a category this is not possible
you might think Aha I've seen this before when I have categories all I need to do is one hot encode
them.
So for example what you would do is create an array which has length equal to the size of our vocabulary.
In other words the number of words in the English language.
Typically we denote that with a capital letter V.
So here's how we are going to 1 Hot in quote a word.
First we're going to create a vector of size Big V containing all zeros.
Then we're going to create a mapping where we have for each word a corresponding integer starting from
1.
So for example the word a we'll get the index 1 The word A will get the index to the word activation
might get the index 100.
And we do this for all other words.
So we get all the way up to Zu which maybe has the index 1 million finally.
Once we have a map for each word to an integer we simply set that corresponding index to 1.
So as you can see all of these vectors are a bunch of zeros with only a single one add a unique position
so thus our final one hot encoded word vector becomes a vector of mostly all zeros except with one one.
So what happens after we do this.
Well now we can go back to our usual scenario if we have a sequence of T words and each of them becomes
a vector of size V then we have a t by V matrix which represents our sequence.
This is the same shape as our generic t by D sequence.
We can then pass this into our recurrent neuron that work as normal.
So let's do a very simple example.
Suppose we had the sentence I like cats.
This is a sequence of length 3 because the sentence has three words.
This might turn into the sequence of vectors 0 0 0 1 0 1 0 0 1 0 0 0.
For this simple example we assume that our vocabulary only contains four words which of course is much
smaller than a real English language dataset
but there are some problems with this approach.
Some English language data sets have about 1 million possible tokens or words that would lead to an
extremely large one hot and feature vector which means that your weight matrix would also be very large.
So in other words you have a t by V matrix but V is a very big number.
In actuality the English dictionary contains only about two hundred thousand words so it really depends
on how you split up your words and where you obtain your vocabulary from.
But this is still a significantly large number.
Here's another problem with this approach.
Remember that we would like our input features to have some structure.
If you recall one of the earliest rules we encountered was machine learning is nothing but a geometry
problem.
This rule relies on the fact that there's geometrical structure in our data.
Data vectors from the same class are probably close to each other
but what do one hot encoded feature vectors look like.
Well if we take any to one hot encoded feature vectors and we calculate that Euclidean distance we will
always get the square root of 1 squared plus 1 squared which is just the square root of 2.
It doesn't matter which two words we select because they are all one hardcoded.
For example the distance between Cat and feline would be square root of 2.
But the distance between Cat and airplane would also be squared of 2.
In other words this data has no useful geometrical structure.
So what can we do instead.
Well it would be nice if for each word we could map them to a D dimensional vector.
Luckily we have just the tool for this.
It's called an embedding layer.
Now you might be wondering how can we actually make these vectors have a useful structure.
We'll get back to that later.
So stay tuned.
But before we discuss that I want to show you a little coding trick
so consider what happens if we multiply a 1 Hot encoded vector with a weight matrix.
Let's suppose the vector is just three dimensional 1 0 0 and the weight matrix just contains the numbers
1 up to 9.
You can verify for yourself that when we multiply this vector by this matrix we get the vector 1 2 3
now consider the one hot and quoted vector 0 1 0.
What happens when you multiply this vector by the same weight matrix.
We get the vector four 5 Six
now consider the 1 Hot encoded vector 0 0 1.
What happens when we multiply this vector by the same weight matrix we get the vector 7 8 Nine.
So what is the pattern we see if the index in the one hot encoded vector which was set to 1 was 1.
Then we get the first row of the weight matrix.
If the index was set to 2 then we get the second row of the weight matrix.
If the index was set to 3 then we get the third row of the weight matrix.
In other words if we won hot in code the integer k and multiply it by the weight by critics.
All that's really doing is selecting the K ith row of the weight matrix.
So here's a shortcut.
The old way of doing this took two steps.
First we had to create a 1 Hot encoded vector and set the K of entry to 1.
Second we had to multiply this one hard and coded vector by the way matrix the shortcut way of doing
this takes only one step.
We simply index the weight matrix at the rock.
This is obviously much more efficient than creating a one hot vector and then doing matrix multiplication.
Think of it this way indexing an array is all of one that's constant time.
But how long does it take to create a one hot vector and then do matrix multiplication.
If your vector is of size V and the weight matrix is a size v times D then this is o of V times D.
So this is exactly what the embedding layer in PI torch does.
Instead of doing all the work yourself to create one hot and coded vectors the only step you need to
do is map your data set with sequences of words into a data set with sequences of word indices.
Now that you have only integers you can use an embedding layer to map each word's integer to a corresponding
word vector.
From there on out your sequence becomes A T by D a matrix at which point you can use an R and then as
you normally would
Conceptually you can think of the process like this.
First we have some sentence a sequence of words let's say for simplicity's sake it's I like cats.
Then this sequence of words becomes a sequence of integers.
Obviously it doesn't matter which integer we assign to each word as long as they are unique.
This is just like when we are doing classification.
It doesn't matter if we set dog to 1 and cat to 0 or vice versa.
They are just arbitrary assignments.
Finally we can use these integers to index an embedding matrix which will convert each integer into
a word vector.
So in the end what we have done is two steps.
First we converted words to integers.
Second we map those integers to vectors
one question you might have at this point is we know intuitively that similar words should be closer
together than dissimilar words.
In other words these feature vectors should exist somewhere meaningful relative to each other.
For example Kings should be close to Queen in car should be close to automobile but Queen should not
be close to car and King should not be close to automobile so how can we make sure that that's the case.
Luckily this is the same story as with convolution of filters because they are just waits in a neuron
that work.
They all get trained automatically.
There is nothing fancy you need to do.
Now there's one caveat to that which is that what people often do is use pre trained word vectors.
How this works is when they create an embedding layer.
They set the weights to a set of pre trained vectors that were obtained through some other method then
they freeze this layer so that those weights are never changed.
Typically these word vectors are found through algorithms such as words effect and glove.
Now that is a deep topic in itself.
So in this course we are going to avoid discussing that.
Instead we will train embedding is like any other layer by letting gradient descent do its magic if
you want.
You're encouraged to read the words of in glove papers since they are generally pretty accessible.
To summarize this lecture here's what we discussed.
First we recognize that it's not possible to multiply a word by a way matrix.
There is no concept of multiplying words.
Therefore we needed to somehow convert words into a numerical representation we first thought about
one hot encoding each word.
But we soon realized why that would not be a good idea.
First of all it would take up way too much space.
There were around 1 million possible tokens in the English language so each word would take up 1 million
numbers to represent which is quite silly.
Also these one had encoded vectors are actually meaningless in terms of being feature vectors.
This is because each word vector if it's just a one hot encoded vector would be equal distances apart
so cat is the same distance the feline as it is to airplane.
So that's not very useful.
It doesn't allow us to really make use of the rule.
Machine learning is nothing but a geometry problem.
And thus we need a better way to convert words into vectors.
This method is to take each word turn it into an integer and then use that integer to index an embedding
matrix.
This is much faster than one hot encoding and then doing matrix multiplication
in this way.
An input sentence consists of words then becomes a t by the matrix of numbers which you know and RNA
can readily accept.
We saw that this layers parameters are trained just like any other layer.
So nothing other than gradient descent needs to be done.
Alternatively if you were interested you're welcome to read up on words of Vec and glove which provide
different ways of initializing word and buildings so that you don't need to train them along with the
rest of the neuron that we're.