So in this lecture, we are going to discuss yet another modification of the basic counting method,
the motivation for this lecture is similar to our motivation for using subir tokenization.
As you recall, with basic tokenization and counting each word, no matter how similar, will occupy
its own component in our final vector.
So if we have the words walk, walking walks and walks, they would all be treated as separate words.
In this case, a walk is as close to walking as it is to cartwheel, which is definitely an unrelated
word.
Another problem with this approach is again related to high dimensional vectors, despite the fact that
these words have very close to the same meaning we treat them all as separate vector components.
Since, as you recall, there is one component per word.
By having separate vector components for many related words, we increase the dimensionality of our
vector representation, which is not good for many reasons, especially when those components may be
correlated.
Yet another reason we don't want to treat these words separately is very practical.
Suppose that we are building a search engine like DuckDuckGo.
Now suppose that I'm searching for content on people running, but content creators might express this
in different ways, they might say.
Here is a picture of me running or this is me when I ran to the park last week, or I run to the office
on Monday.
All of these are relevant search results, but they wouldn't be considered a match if we treated each
variation of the word run in a different way.
So what is one possible solution to this issue?
The main motivation of this lecture is to take these related words and simply convert them to the same
word as a pre processing step that is walk, walking walks and so forth.
All become walk in some sense.
What we want is the root word.
This lecture will look at two of the most popular techniques for doing this, known as stemming and
limitation.
So what's the difference between stemming and limitation, the difference between a stemming and limitation
is that stemming is very crude and won't necessarily output real words.
It simply chops off the ends of a word and gives you back the rest.
On the other hand, limitation is more sophisticated, and it uses actual rules of the language in order
to give you back the true base word or route.
The route is also known as the lemma, hence the term limitation.
Either way, both of these methods will help you reduce your vocabulary size, which will also reduce
your vector dimensionality and hence speed up any further processing you might need to do on your text.
So let's start by looking at stemming, as mentioned, stemming is a very simple method based on heuristics.
For example, if a word ends in yes, remove the final yes, so bosses becomes boss or if a word ends
in EMEA a. Remove the whole thing, so replacement becomes plaque without an e at the end.
Note that this is not a real word.
Also, note that stimming is not just one method.
There are multiple kinds of steamers.
The most popular algorithm is the Porter Steamer, which can be found in Nulty K and other libraries
as well.
So if you'd like to use it, here is some sample code.
First, you want to import the Porter Steamer class.
The next step is to create an instance of the Porter Steamer.
At this point, you can pass in any token you want to stem.
For example, if I call Port, I'm walking, this will return the string walk.
Now that we've discussed stemming the next step is to discuss limitation as mentioned, limits ization
is a bit more sophisticated.
Basically, you can think of this as a lookup table or, in other words, a dictionary as an example,
suppose we want to find the root of the word better if we were to use a steamer.
It would simply return the same string better.
But if we use a limited user, it will return to good, which is a completely separate word in terms
of how it's spelt.
There's no way to manipulate the string better to become good.
It simply must come from a database of knowledge.
Another example are the words was and is the word was, is the past tense of is.
But these are both derivatives of the word be a steamer will take the word was in return water without
the yes, but a limited zero will return be in both case, it's.
Another nice example is the word mice.
This is one of those strange words that is a plural, but it doesn't simply add an s at the end of the
original word.
A steamer will simply return the same word mice, but a limited zero will return the word mouse.
So note that limited visitors appear in many Python libraries, such as Nulty Can Spacey, we'll look
at NCTC since the API is a bit simpler and mirrors that of the of cinema.
So again, we begin by importing and I'll take in the word net limitées a class.
The next step, if you haven't already done so, is to download the word in that database, which the
word net limitation makes use of.
Once we create an object of Typekit word net limits area, we can then call its function limits to convert
one word at a time into its corresponding lemma.
One important thing to recognize is that although it's not required, the limits function it takes in
one extra argument called P.O.S., which stands for parts of speech.
Basically, this signifies whether the word is a noun, verb, adjective and so forth.
The default is noun, but note that this does not work for all cases.
For example, if you pass in the word going by itself, the limiter will return to going, which is
not what we want.
Also note that the word going is not a noun.
Instead, we should specify that it's a verb, in which case the word go will be returned.
So although the parts of speech argument is not required, it is strongly recommended to make use of
it.
So why might we want to do this where we do something different based on the parts of speech?
Well, let's consider this sentence.
Donald Trump has a devoted following.
And let's consider another sentence.
The cat was following the bird as it flew by in the first case.
Following is a noun, in which case that is its root form.
In the second case, following is a verb, in which case its root form is followed.
Thus, we have seen that the root form of a word can be dependent on its parts of speech.
Now, there is one weird quirk about Nalchik, which essentially just wraps another library to do limited
zation, as we just saw in order to properly use the word in limited Heizer.
We need to first do parts of speech tagging on our sentence.
So you may have assumed that you can just use an LG K to do parts of speech tagging, which you can
and then you can pass in those parts of speech tags into the limitées function.
But unfortunately, the parts of speech tags, which are returned by analytics tagger are not compatible
with the limited size function.
Instead, they both use different sets of tags.
And so one way to get them to work together is to map the tags from the parts of speech together into
a form that's acceptable to the limited size function.
We'll go through the exact details in a notebook, but this is something that you have to know if you're
going to use the limits of correctly.
Now, the final topic I want to discuss in this lecture is the application of stemming and limits ization
to real world scenarios.
You may believe that these techniques are outdated because you can just leave all your words as is,
and use a deep neural network to do any NLP task, but you would be incorrect.
This is due to a bit of shortsightedness and lack of awareness of where NLP is used in the industry.
So if you think that generating text like Djibouti or doing sentiment analysis and spam detection are
the only applications of NLP.
This is simply not true.
So here are some billion dollar industries where these techniques are used.
Some examples are search engines and document retrieval.
Online ads and social media tags.
As an exercise, I recommend thinking about how stemming in limited physician could be applied in these
scenarios.
The first example, which I briefly mentioned earlier, is search engines.
In fact, searches how Google became one of the largest tech companies in the world.
As you recall, when a user enters a query, we don't want to only include exact matches because then
we wouldn't get back all the relevant results.
Instead, by converting all the terms into their route form, we can search through more possible matches.
Also, keep in mind that the users who enter these search terms in the first place won't answer them
the same way, even though they might want the same thing.
So one person might type running while another might type Iran.
But the meaning of these two words is essentially the same.
We want the search engine to respond to what the user means, not necessarily what they typed.
Furthermore, we want to make things easy for our users.
Imagine if the user had to type in all the different variations of each word run.
Running runs ran and so forth just to get all the relevant results.
That would take a lot of time, and users would not want to use a search engine that requires them to
do so much work.
Here's another example, which is closely related to Google search.
And this is online advertising.
In fact, although you may know Google because of its famous search engine, it is in fact mainly an
ad company, as that is where it makes most of its revenue.
So how do ads work?
Well, ads are all based on keywords.
You can think of keywords as the relevant terms that a user types into the search box.
If you are an advertiser, then you want to match your ads to those search terms.
For example, suppose you are working for Apple and your job is to run ads for the iPhone.
Now, let's keep in mind that you have to pay Google every time they show one of your ads.
So do you want Google to show your ad when someone typed send flowers or a Tesla Model three?
The answer is no, because iPhone is irrelevant to those search terms.
Instead, you want your ad for the iPhone and to show up when users are searching for phones or other
related items.
So perhaps when they are searching for the Google Pixel or a Samsung Galaxy, that is when you want
to show your ad.
Of course, these keywords can have many variations.
Again, imagine you are selling or running shoes.
You don't only want to cover the word running.
You also want to cover words like run runs and so forth.
So again, stemming in limited vision are useful in the scenario.