Here's the code to encode
the two sentences that we just spoke about.
Let's unpack it line by line.
Tensorflow and keras give us
a number of ways to encode words,
but the one I'm going to focus on is the tokenizer.
This will handle the heavy lifting for us,
generating the dictionary of
word encodings and creating vectors out of the sentences.
I'll put the sentences into an array.
Note that I've already capitalized
'I' as it is at the beginning of the sentence.
I then create an instance of the tokenizer.
A passive parameter num wards to it.
In this case, I'm using 100 which is way too big,
as there are only five distinct words in this data.
If you're creating a training set based on lots of text,
you usually don't know
how many unique distinct words there are in that text.
So by setting this hyperparameter,
what the tokenizer will do is take
the top 100 words by volume and just encode those.
It's a handy shortcut when dealing with lots of data,
and worth experimenting with when you
train with real data later in this course.
Sometimes the impact of
less words can be minimal and training accuracy,
but huge in training time,
but do use it carefully.
The fit on texts method of
the tokenizer then takes in the data and encodes it.
The tokenizer provides a word index property which
returns a dictionary containing key value pairs,
where the key is the word,
and the value is the token for that word,
which you can inspect by simply printing it out.
You can see the results here.
Remember when we said that the word I was capitalized,
note that it's lower-cased here.
That's another thing that the tokenizer does for you.
It strips punctuation out.
This is really useful if you consider this case.
Here, I've added another sentence,
'You love my dog!'
but there's something very different about it.
I've added an exclamation after the word 'dog!'
Now, should this be treated as
a different word than just dog?
Well, of course not.
So the results of the code that we saw
earlier with this new corpus of data,
will look like this.
Notice that we still only have 'dog' as a key.
That the exclamation didn't impact this,
and of course, we have a new key for
the word 'you' that was detected.
So you've seen the beginnings of handling texts by
creating word-based encodings of that text,
with some very simple code intensive flow and keras.
In the next video, we'll take a look
at the code and see how it works.