So in this lecture, we are going to discuss one of the details we previously glossed over, which is
how do we actually get the words from a string of multiple words?
Now, if you've ever coded in languages like C or C++, you may recall having to write the code to do
this yourself.
Luckily, in the higher level languages like Java and Python, this is not the case in Java or Python.
You simply need to call the string function named Split.
What this does is is it takes a string containing multiple tokens and another string to separate each
token by, for example, a space or punctuation or any other character or sequence of characters.
The output of this call is a list of strings where each string in that list is a token.
For example, if they pass in the string, I like cats and I call the split function.
This will give me back a list with three items I like and cats.
Note that by default, the split function will split on whitespace.
So this process is what we call tokenization.
Unfortunately, this is somewhat of a confusing term, you see in the old days when we said it tokenization,
we really meant what we referred to in modern times as a string split.
So if you look at tutorials in C and C++ and they talk about tokenization, that does the same thing
as the split function in Python today, when we talk about tokenization, it can be generalized to include
many more features.
The final note I want to mention on this slide is that you can view this lecture from two different
perspectives.
One way to view this lecture is to learn about all the different tokenization options you could apply
if you use the site could learn a count of activities or class in this case.
Tokenization is actually done for you, and so your job is to simply select the right options for your
data and your task.
On the other hand, if you want to implement count of factorization yourself, then you should view
these as possible steps you could take in your implementation.
Luckily for many applications, these extra features we're about to discuss are not needed, and thus
it's completely fine to think about tokenization as simply being a string split.
For example, if we're using deep learning and we want to classify text with an aunt, then just using
a string split turns out to work pretty well.
But there are several ways that we can expand the functionality of a tokenize here.
So in this lecture, we'll be discussing all of these possible varieties of tokenization.
The first thing I want to mention is punctuation.
Punctuation may be important for downstream and Ielpi tasks.
For example, sentiment analysis.
Suppose we have a sentence I hate cats, which ends in a period.
Now suppose we have a sentence.
I hate cats, which ends with a question mark.
These two sentences clearly have different meanings.
The sentence that ends in a period is a statement declaring that I dislike cats very much, but the
sentence that ends in a question mark is a question.
It could mean that the person asking the question is wondering why someone else thinks that they hate
cats.
So in this context, punctuation matters, and thus it is possible, in some token, ICER's to treat
punctuation as tokens as well.
So both of these sentences would be converted into four tokens where the token is the punctuation.
Now, do you always need to do this?
The answer is no.
It really depends on the results you observe.
Use it if it works.
Don't use it if it doesn't.
By default, the count victimizer insight can learn completely ignore as punctuation.
So that was one way to deal with punctuation.
But here's another way that could be more simple.
As you recall, the simplest way to do tokenization is to just call string split.
Note that if you do the simplest thing, which is to split on whitespace, this will keep the punctuation
along with the word itself.
That is, I hate cats.
The statement would still be treated differently than I hate cats.
The question The only issue with this is you may need a lot of data for the model to learn from.
That is to say you'll want a sufficient number of sentences that end with cats period and a sufficient
number of sentences that end with cats question mark.
As you recall, for machine learning data is what we need to make our models learn.
And so the more examples we have, the better.
But by keeping punctuation with the words when we do a string split, this effectively increases our
data requirement.
Now, the reason I mentioned this is not to suggest to you what to do or not do, but to simply tell
you the pros and cons of each approach.
Ultimately, it's still up to you to check the performance on your dataset.
OK, so the next modification I want to discuss is casing again, suppose we're doing some task like
sentiment analysis or spam detection.
It's probably the case that the word cat, all lowercase has the same meaning as cat with a capital
S..
In other words, we may want our model to be case insensitive.
One way to do this is to simply lowercase all the letters in our text corpus.
Note that with the count, victimizer inside could learn this can be done by setting lowercase to true
when you construct your account victimizer object.
Now again, whether or not this will help is not guaranteed, and you always have to check the results
for yourself.
Yet another modification I want to discuss is accents, as you recall, some words, although it is
less common in English, have accents.
However, people who speak English tend not to use accents, even if they are using a word that has
accents.
For example, the word naive.
Technically, this word is spelled correctly, whether it has accents or not.
Therefore, both should be treated the same.
If you're quoting this yourself, you could write a function to map accents and characters to the corresponding
non accented character.
Or if you're using psychic learns count of exerciser, then you can simply use the argument strip accents.
OK, so throughout this lecture, we've been assuming that our tokens are full words.
However, this need not be the case.
In fact, it's also possible to build models based on characters as well.
And in fact, we will do so in this course.
Now, this is just a brief preview, but let's go through some pros and cons of different approaches
as you learn more about different techniques, such as deep learning.
The reasons why these are pros and cons will become more apparent.
So let's think about word based tokenization.
As mentioned, there could be up to a million words in a vocabulary because of this when you're building
a model.
You actually need to store all these million words.
Or if you want a vector representations of these words, then you need up to a million vectors, which
could take up lots of space.
Furthermore, in deep learning, if we're going to build some kind of language model, then our neural
network will output a probability distribution over all the words in our vocabulary.
You might not realize why this is bad now, but intuitively you can imagine that a probability distribution
with one million possible values might be hard to get exactly right.
If you know a bit about deep learning, then you know that this would require a final weight matrix
with size one million in the last I mentioned, which is pretty large by deep learning standards.
So these are some disadvantages of using word based tokenization.
On the other hand, words do contain a lot of information.
For example, when I see the word cat, this has meaning we can picture this in our minds.
It's pretty much unambiguous what a cat is.
Now consider character based tokenisation characters, unlike words, do not contain much information.
Take the letter C.
I know that the word cat contains the letter C, but so does the word car.
So does the word carbon.
And these are completely different things.
So the letter C does not tell me a whole lot about the idea I want to represent.
So that's a disadvantage of using character based tokenization.
On the other hand, in English, there are only 26 letters.
Add to that a few different whitespace tokens and some punctuation.
And there really aren't that many characters.
Thus, if we use characters, then our vocabulary size is small, much less than one million, and it
is easy and efficient to represent them in a computer.
OK, so these are two options for tokenization.
Word based or character based?
Both have pros and cons, and neither is perfect.
Now there is a third tape of tokenization, which is sort of a middle ground between word based and
character based.
This is sub ward based tokenization.
In this case, some words can actually be split up into multiple sub words.
For example, take the word walking walking can be split up into two sub words walk and doing.
So why might we want to do this?
Well, consider the fact that the word walk by itself is closely related to the word walking.
Thus, we would want them probably to have some shared representation in our machine learning model.
You can think of the suffix inji to simply be a modifier on the word walk.
On the other hand, suppose that we did not tokenize walking into two sub words, what would happen
is walk in walking would be treated like two completely different words.
In this case, walk is no closer to walking than it is to a tree, which is a completely unrelated word.
If we did this, then our model would have to learn the relationship between a walk and walking using
only the sentences we pass in.
We could only hope that the model realizes they are similar.
If we do split them into sub words, then a model knows that the subway to walk appears in all cases,
whether it is walk walking, walked walks and so forth.
So the question is, do we want our model to know that the same subway ride appears whenever any of
these words appear?
Or do we want our model to have to learn about each variation independently?
Now, I realize this makes a strong case for Subway tokenization.
And yet we pretty much won't encounter it in NLP until we study Transformers.
What you'll see is that although Subway tokenization seems to be a pretty powerful middle ground between
word based and character based methods, it's not necessary to build a good model.
Now, in terms of implementation psyche, it learns that count victimizer allows you to do both word
based and character based tokenization.
We can control what it does by using the analyzer arguments in the constructor.
Specifically, we set this to a word.
If we want word based tokenization or car, if we want character based tokenization, note that the
default is word.
Now, if you want to implement tokenization yourself, then note that a string is already a sequence
of characters.
So if you had a string s and you did a for loop for C in s, then this would give you each character
one at a time and thus you wouldn't have to call string split.
You would simply use the string as is perhaps after lower casing and removing accents and so forth.
Now, the final topic I want to discuss in this lecture is how is tokenization actually done?
I want to mention this because in my first and Ielpi course, I had a lot of beginners get angry with
me for this.
The beginners thought that there was some fancy machine learning algorithm that they had to learn in
order to understand how tokenization works.
So they thought I was withholding information from them or failing to teach them something.
But of course, this is incorrect.
This is one task in NLP that does not require machine learning.
As mentioned, tokenization can be as simple as a string split, and if you want to know how that works.
Recall that there are many tutorials out there in C and C++ to explain it is just basic coding.
In fact, if you can't imagine immediately how it would be done, I would actually recommend that you
try it as an exercise.
OK, but what about the more complex types of tokenization like what we mentioned in this lecture?
Well, again, note that none of these involved machine learning in any way.
Lowercase letters and removing accents are just mapping one character to another character.
OK, but what about punctuation and subway tokenization?
Well, let's remember that languages essentially follow a fixed set of rules.
There is nothing to be learned.
And in fact, there are many exceptions as anyone who has studied language should know.
For example, there's a rule in English that says I before e except after C as an example of a word
that follows this rule, consider the word receive.
But there are many exceptions to this rule.
For example, the word weird.
So if you're trying to follow patterns, which is what machine learning does, then you would likely
fail because exceptions are the opposite of patterns.
So in fact, it's not machine learning that we need for tokenization, but simply a rule based routine
that takes into account all of these exceptions.
OK, so I hope that clears up any misconceptions you may have had about string slips.
It's just basic programming, and it is not machine learning.