So in this lecture, we are going to discuss one important topic which will be necessary if you want
to implement DFAT from scratch.
Doing so would be considered an advanced exercise.
So if you'd like to try this exercise?
Note that this is a step that must be completed if you only want to use.
I could learn that you don't necessarily need to know how this works at the same time.
Note that if you want to do anything more advanced in NLP, you still have to know how to do this anyway.
For example, if you're going to do NLP and TensorFlow PyTorch or Ajax, you'll need to know how this
works.
So methods like words of Vec aren't ends and transformers will all require some knowledge about this
topic.
OK, so what is this topic, the topic of this lecture is the word to index mapping.
We'll discuss what that is, why we need it and how to build one from a given text corpus.
OK, so let's start by answering the question, why do we need the word to index mapping as you recall
what we do when we convert text into vectors as we take a whole set of documents and convert that into
a matrix?
Essentially, each row of the matrix is a vector, or each vector corresponds to one of the original
documents.
Each column of the matrix and hence each component of the vector corresponds to a word.
Thus, the total size of the matrix is a number of documents by a number of words.
The big question is this How do we know which column corresponds to which word?
This is not obvious.
Perhaps you may believe they will be in alphabetical order.
Perhaps you may believe they will be in the order the words appeared in in the text corpus.
Perhaps you believe they may be ordered by the frequency of those words in the text corpus.
Of course, none of these is necessarily false, but none of these is necessarily true.
The truth is, it depends on how you decide to build the word to index mapping.
That is, you get to decide which column corresponds to which word.
This lecture is mostly about how you can actually write the code to do this.
Now, I have to warn you that this is one of those tasks which really requires you to know how to program
if you can code and you can logically think through an algorithm, you should find this to be quite
easy.
If you cannot code, you may find this to be difficult for many Udemy students.
I've seen them struggle with this in the past.
My best advice for you is to simply practice coding and improving your skills.
This is one of those things where it's just pure coding.
There are no tricks and there is no specialized knowledge you need.
It's just basic coding, and it only depends on your ability to think through what you have to do.
In other words, your ability to get this done is dependent only on your thinking ability.
OK, so as a simple example, we're going to start by looking at a text corpus that we can process by
hand.
Let's suppose our text corpus contains only three documents.
Document number one is I like cats.
Document number two is I love cats.
Document number three is I love dogs.
OK, so our first order of business is to determine what is our vocabulary size.
That is how many unique words exist in our corpus.
In this case, we have five words.
I like cats, love and dogs.
Thus, each of these words would occupy columns zero, one, two, three and four inside a matrix with
five columns.
So, for example, we might say zero corresponds with I.
One corresponds with like two corresponds with cats and so forth.
In this case, the integers have been assigned in the order that each of the words appears.
You'll see that this is quite natural when we actually look at some code that will do this.
Now, this example is really easy because it's small enough that you can do everything in your head.
The question is, how can you do this when you have millions of documents and thousands of possible
words?
The answer is you're going to have to write some kind of computer program in this lecture, I'm actually
going to give you some pseudocode that will accomplish this task.
You don't have to follow this particular strategy, but this is just one possible way to make a word
to index map.
If you have your own ideas about how to implement this, you are strongly encouraged to try that instead.
OK, so let's look at our pseudocode.
Note that while I call this pseudocode, it's really actual Python code that will work given that your
inputs are formatted correctly.
So although this is pseudocode, it is also real working code.
OK, so how does this work?
Well, we start by initializing a variable called current index to zero.
We also create an empty dictionary called word to index.
Now, recall that when we say mapping in Python, this usually translates to dictionary mapping is where
we take one value as input and produce one corresponding value as output.
That is, the same input should always map to the same output.
And this is exactly what a Python dictionary does.
The next step is to live through our documents.
This is assuming that our documents are stored in a list or some editable in each individual component
is a single string.
The next step is to tokenize the document, as you recall, we've seen several ways of doing this,
such as a simple string split or more complex methods like notecards where tokenize.
This will give us back a list of tokens, which we will assume are words.
Note that this may contain punctuation as well, but for the purpose of this lecture, they should all
be treated the same.
The next step is to move through each token inside the loop.
We check whether or not the token exists in our word to ADX Dictionary.
If it does, then there is nothing to do.
That means our token has already been mapped to an index.
The interesting case is when the token does not yet exist, in this case, we must create a new entry.
We can do so by setting the key of our dictionary to be the current token and the corresponding value
to be the current index.
We must also remember to increment the current index so that the same variable can be used in the same
way.
The next time we encounter a new token in this way, we will increment current index one by one.
In the first token we encounter will be assigned index zero.
The second token we encounter will be assign index one and so forth.
Once this loop is complete, we will have encountered all the unique tokens in our corpus and our word
to index mapping will be populated correctly.
Now, at this point, once you know which word maps to which index, you'll be able to do things like
implement your own count vectors or your own at TFI Taf in the next lecture, we will be implementing
TFI Taf from scratch, which involves all three of these steps.
As you'll see, creating your own town vectors requires you to first create a word to index mapping
and creating your own TFI.
Taf requires you to first create your own count vectors.
Thus, every step we've learned about in the section must be completed in TFI.
TAF is like the culmination of everything we've seen.
So for those of you who might be thinking a few steps ahead, you may be wondering how do we treat words
which appear in the test set but do not appear in the train set?
The answer is that there are several ways to deal with this.
Method number one is to simply ignore words which were not seen in the train set.
This makes sense.
For example, if you're building a machine learning model, your model will be trained on the train
set by definition.
Therefore, it won't know what to do with inputs it has never seen because it hasn't been trained to
do so.
There are some situations, however, where ignoring words is not desired.
Method number two is to create a special index for unknown words or rare words that did not appear in
the train set.
In this case, even some words in the train set could be considered rare words such that they get mapped
to the special token.
We could, for example, assign any words whose frequency falls below a certain threshold to be an unknown
word.
In this way, our model will learn what to do with unknown words since they appear in the train set
and then during test time, it will handle new unknown words in the same way.
The downside to this is that all unknown words will be treated the same.
The next topic I want to discuss in this lecture is the reverse mapping that is the mapping from index
back to work.
So why would we ever want this?
One example is with neural networks.
As you recall, machine learning works with numbers.
One way to use neural networks is to take a list of words in a sentence and to try and predict the next
word.
The next word, however, is represented by a number in the neural network output.
So the neural network might say the most likely next word is word number 500.
Of course, in order to figure out what this word actually is in human readable terms, we must know
which word corresponds to word index 500.
Thus, we need a mapping from index back to word.
Another example is with interpretation of machine learning models.
Note that this could be deep neural networks, but does not necessarily have to be.
So in machine learning, we often like to know which of the input features is most important.
This helps us to interpret what our model has learned.
So our algorithm might say input number three, four or five is the most important input, but how do
we know which word input number three, four or five corresponds to?
The answer is we must keep track of this.
And the way to do that is with a mapping from index back to work as an exercise.
You should think of how one might convert a word to index mapping into an index to word mapping and
vice versa.