So in this lecture, we'll be looking at how to do tax preprocessing using the Keros API.
Note that this is an absolutely critical.
In fact, what we'll be doing is looking at the syntax to do something which you've already learned
how to do.
The reason we're doing this is because Keros makes it easier to do these steps.
In fact, the API is so nice that sometimes PyTorch users will just use crisp processing instead of
PyTorch, although this may be due to the fact that the PyTorch text library is pretty bad in any case.
Do note that while this does make things easier, this comes at the cost of some flexibility.
For example, the Index zero is reserved for padding, so you won't be able to assign this to a word,
as you'll see.
This makes the embedding matrix somewhat inefficient because the first row becomes useless.
Consider it an exercise to think about why this is the case after you've learned about embeddings.
So before we move on to the TensorFlow syntax, let's review how exactly we want to convert text into
integers.
This is just very high level, but if you want more detail, simply go back to the previous lectures
in the course.
So the first step is to tokenize our text.
Recall that our text will consist of documents, and each document is just a big long string containing
multiple sentences, each of which may contain multiple words.
Our goal with tokenization is to convert each document into a list of tokens.
Each token might be a whole word, part of a word, or just a character.
Once we've tokenized our text, the next step is to assign an integer ID to each unique token.
So the word A might get it zero AA might get one, bobcat might get eighty one thousand and so forth.
Let's recall why we want to do this.
The reason we need an integer to map to each token is because these integers represent the positions
in a feature vector.
For example, if we're building a derivative matrix, we need to know which column corresponds to which
token.
Now, going forward, we won't be using TFI Taf, but we still need to be able to map tokens to integers.
As you'll see, in fact, for most of deep learning, we won't be using bag of words, but instead we'll
be working with full sequences.
To that end, there will be one additional step in this lecture, which is that after we create a mapping
from token to integer, we're going to convert his document into a sequence of integers.
So, for example, the document I like cats might become a list containing the integers four three,
five six, oh two and 27, implying that four three five represents i six 02 represents like and 27
represents cats.
Practically.
You can see how this might be helpful in terms of memory strings.
Take up a lot of space, but numbers do not.
So intuitively, you can understand how passing data around in your code in terms of numbers is much
more efficient than text.
OK, so in TensorFlow, all of what I just described is done using the tokenize or class.
This is imported from the Keras module since it was originally part of Keros before TensorFlow adopted
us.
You'll notice that the interface is kind of like what you see inside.
You learn where you fit on some text and then you transform on other, possibly different text.
Except in this case, the fit method is called the fit on text in the transform method is called text
two sequences.
So this will directly convert any input text into sequences of integers.
As usual, you typically want to fit on the training set well, you only transform on the test set.
This emulates the real world operation of your model since in the future it will need to transform data
your model has never seen.
Now let's talk about what kind of data you should pass into these functions.
Basically, you should pass in lists of strings.
So, for example, if you have 100 documents, then you would pass in a list of size 100, and each
element of that list would be a string representing a document.
But note that the tokenize is pretty flexible.
So alternatively, instead of having each document be a single string, you can also have each document
be a list of strings.
In that case, it would mean that each document has already been tokenized.
So in that sense, the name of this object is kind of misleading because although it does do tokenization,
it also does many more steps, such as creating a mapping from each token to an integer and converting
each document into an integer list.
And even more strangely, it works even when your inputs are already tokenized, in which case the so-called
tokenize are wouldn't actually be tokenizing anything.
So let's talk about what kind of output one might expect after using the methods I just described.
Suppose we pass in a very small dataset with just three documents in each document is a sentence.
The first sentences I like eggs and ham.
The second is I love chocolate and bunnies.
And the third is I hate onions.
After using the code, I just described the output of the text to sequences.
Method would be as follows.
As you can see, it's just a list of lists of integers where each integer corresponds to a word.
Now you might be wondering how do I know which word corresponds to which integer?
Of course, this information is stored within the tokenizing object.
Simply use the attribute word index, and this will return a dictionary where the word is the key in
the integer is the value.
So let's discuss some of the arguments into the constructor.
The first argument is NUM words, which allows you to specify the maximum number of words you want to
keep.
As you've seen, the vocabulary size of any language can get quite large, so this might help your model
focus on only the most important words.
As you might expect, the most frequent words are kept.
The next argument is Filter's, which specifies which characters will simply be removed and ignored
by default.
This includes all punctuation tabs and line breaks, except for the single quote.
The next argument is lower, which allows you to specify that all texts should be down cased, the default
for this is true.
The next argument is split, which specifies the separator for each token.
The next argument is car level, which allows you to specify that you want character tokens instead
of word tokens.
By default, this is false.
The next argument is over a token which stands for out of vocabulary token.
This is for tokens which are not included in the vocabulary.
For example, if you encounter a word in the test set that was not in the train set, it would be assigned
to this token.
If you don't set this argument, these tokens will simply be ignored, which is the default.
Now you'll notice that there's nothing in the arguments about stemming or limitation or stop words,
as we learned about earlier in the course.
And this is because we typically don't want to do these operations in deep learning as an example,
suppose you're trying to generate text.
This would be pretty difficult if you couldn't generate stop words as stop words are pretty common.
So your text would be incoherent.
The same goes for stemming in limitation as an example.
If you're doing machine translation, running in Iran will have different translations.
In this case, the difference is important.
Now, just as a small bonus, I want to discuss another method you can call, which is text to matrix.
Essentially, this allows you to convert your text into TFI Taf count vectors or binary vectors, just
as we described earlier in the course.
This won't be needed since in deep learning, we actually want to keep our text as a sequence, but
it's nice to know that it exists.
Now, at this point, we're going to look ahead a little bit and talk about the concept of padding.
The reason why padding is required is because in some deep learning libraries such as TensorFlow, your
sequences must all have the same length.
Practically, you can see why this might be the case.
We know that we like to use arrays to store our data.
But arrays must have the same size in all dimensions.
There's no jagged array in Nampai, but with text we do have variable length sequences.
Not all sentences have the same length, and not all documents have the same length.
I mentioned that this involves some looking ahead because you won't really see why this is true until
you study CNN's in earnings.
So if you haven't studied those topics yet, you might not have any intuition about why we should do
this for now, just trust that this is true.
In any case, there is a function called pad sequences, which is also part of the Keros API.
The default action is to take your list of lists of integers and to prepend zero so that all of the
sequences have the length of the longest sequence.
So if we use our example from earlier, you can see that two zeros have been added to the front of the
third sentence because that sentence only has three words, whereas the other two sentences have five
again.
Looking ahead a little bit.
It's typically the case that we want to add zeros at the front, since models like aren't ends have
trouble remembering things from too far in the past.
Thus, we would like to have the most useful information and near the end.
There are other options we'll explore in the next lecture, such as adding zeros at the end instead
of the front, and truncating the sequences to limit the maximum length.
As usual in machine learning, it's useful to think of shapes.
So now that we've converted our text into tokens and then into a padded sequence of integers, let's
think about what shape we should have if we were to store the results in a numpy array or TensorFlow
Tensor.
Basically, just remember end by T and is the number of documents and T is the maximum document length.
Just as we use the letter D by Convention for a number of features, we use T by convention for a sequence
like.