In this lecture, we are going to look at a notebook which demonstrates how to do a tax preprocessing
processing that we just discussed.
As usual, you can look at the title of the Notebook to determine what notebook we are currently looking
at.
So to start, we're going to import the class tokenize here and the function pad sequences.
All right, next, we're going to create a dummy dataset with just three sentences.
I like eggs and ham.
I love chocolate and bunnies, and I hate onions.
It's right that.
OK, so next, we're going to define a max of vocab size to be 20000.
Now this is usually pretty reasonable value.
The Oxford Dictionary has almost 200000 words, but the most common words in the English language number
only about 3000.
So three thousand will cover about 95 percent of most texts, according to a quick Google search.
So thus 20000 is probably good enough.
So in the first line here, we instantiate the tokenized class.
Next, we call tokenize airport fit on tax and pass in our sentences list.
Next, we call it tokenize are tax two sequences and pass in the same sentences list.
This returns our sequences of integers.
By the way, you can think of these two functions in terms of psychic learns feature transformers where
you would always have a dart fit and dart transform.
So the first one is like a dot fit, and the second one is like a dart transform.
Same idea.
So let's run this one.
Next, we're going to print out our sequences list.
Now, normally on a text data set of any practical size, this would just print out way too much data.
But since we only have three sentences, this is reasonable.
So let's run this.
OK, so here we can see that each sentence has been converted into a list of integers.
Each integer corresponding to a word.
Importantly, note that the integers start counting from one and not zero, as you may have expected.
As I mentioned earlier, this is because TensorFlow uses zero for padding.
And you might be wondering, how do I know which word corresponds to which integer?
Luckily, the tokenize are objects already stores a dictionary with this information.
So let's print out the tokenize, their dot word index.
And this is just like our word to index mapping, as we discussed in the intuition like here.
So you should be able to look at this and confirm that if you map each integer back to the corresponding
word, you should get back the original sentences.
Next, let's try out the function at pad sequences in the first block here, we're going to call pad
sequences with all the default values.
So what do we get?
Well, the default appears to set the maximum sequence length to be the maximum sentence length, so
that's five.
The first two sentences were already length five, so there's no padding for the third sentence.
Padding was added at the beginning.
Now in the second block, we can see that if we explicitly pass in Max land equals five, we get the
same answer.
Next, let's see what happens if we start padding to post.
So we can see that the first two sentences still have no padding because they are of the maximum length,
the third sentence now has zeros at the end instead of at the beginning.
Next, we can see what happens if we add too much padding.
So let's set padding equal to six, which is longer than all of the sentences in our dataset.
As you can see, the first two sentences have been padded with one zero so that the sequence length
is six.
Next, we're going to see what happens if we set Maslen equal to a number less than the maximum sequence
length.
So let's try it for.
In this case, we can see that each sequence has been truncated to cut off the beginning of the sequences.
This makes sense as the default because an aunt N. typically pays more attention to the final values
in a sequence anyway.
Next, we're going to see what happens if we set max lenses for again, but this time we're going to
set the truncating argument to post.
All right, so in this case, we can see that the ends of the sequences have been cut off instead of
the beginnings.