So in this lecture, we will be doing a short demo in CoLab to see for ourselves some of the differences
between us stemming in Limited's Asian note that many of these are the same examples we've just discussed.
So if you feel you already have a good understanding, then please feel free to move on.
This lecture should help draw the points home, since this is real code and you don't have to believe
anything on faith.
OK, so we'll start by importing, and I'll take note that in this notebook, we will be importing things
as needed.
But in practice, your imports will typically go at the top.
So just be aware of this stylistic difference, which is only for the purpose of this lecture.
The next step is to import the Porter Steamer class.
Now that we have the Porter Steiner class, we can create an instance which we will call Porter.
The next step is to use the Porter Steamer to stem the word of walking.
As you can see, the STEM for this word is walk.
Now, let's try walked, which is the past tense version of Walk.
Again, we get walk, which is what one might expect.
Now, let's try walks, which is the plural of walk.
So again, we get walk.
Now, let's try something a bit more complex.
The word RAM is the past tense of run.
OK, so as you can see, there is no rule for this particular sequence of letters.
The result is still ran.
This could be an issue since one might think of the root word as run.
Now, let's try running.
OK, so in this case, we get run, which makes sense.
Now, let's try the word bosses, which is a plural, but ends with ease instead of just as.
OK, so as you can see, if this turns out to give us back a real word, which is the singular form
boss.
Now, let's try the word replacement.
OK, so this time we do not get back a real word.
OK, so in this next block, we are going to stem an entire sentence limitation is more sophisticated
than stemming.
So one thing to be mindful of is that the Porter Steamer stems a single word at a time, so if you have
a sentence of words, you don't pass in the whole sentence.
But instead, you split the sentence into single words first.
As you can see, we've called the split function to do just that.
The next step is to loop over each token in the sentence and then call the STEM function on each token.
We'll also print out each stem ending with a space so that the entire printout fits in a single line.
OK, so as you can see, several of the words in the above sentence have been transformed.
We can see that limitation sophisticated and stemming have all been made into a simpler form.
Again, a note that not all of these are real words.
So here's one interesting rule in the Porter steamer.
Let's see what happens if we stem the word unnecessary.
So as you can see, the word didn't get shorter, but the final why was replaced with and I also note
that the word on in front of the word was not removed.
This would make sense in terms of doing an LP since although the word necessary is kind of the root
of the word unnecessary.
They have opposite meanings, so we wouldn't want to treat them the same way in any downstream task.
In other words, it helps us that they would remain distinct words.
Let's try this again on the word buried.
So again, we see that the final why is replaced with an eye?
The next step is to import the word in a limited isare.
Now, as you recall, limits ization essentially amounts to looking things up in a database.
Our data in this case is part of Nutcase Word Net Package, which we can download by calling NCTC download.
So you might be wondering what would happen if we did not include this code.
For example, if you forgot to include it, this is entirely possible, by the way.
For example, if you installed in a long time ago, you'd probably just forget.
And then when you install your code on a new machine, you'll get an error which will remind you to
download this database.
Now, luckily, analytics is pretty smart, and it knows what you need to download when you call certain
functions.
So suppose you forgot this?
Well, then you get an error saying to run this line in effect, you don't have to worry too much about
forgetting this, because Ntsiki will remind you anyway.
So the next step is to import the warden and module from Knowles-Carter Corpus.
We'll see how this is used very shortly.
The next step is to instantiate an object of type order that limits her.
Now that we have our limited zero, we can test it on words.
So let's start with walking as we did before.
OK, so interestingly, the result is still walking, as you recall.
This is because the limits function takes in an argument for part of speech, which by default is noun.
Walking is not a noun, and this is why the word remains unchanged.
Now, let's try walking again, but this time, let's correctly specify the part of speech as a verb.
Note that these are constants which come from the word that module.
OK, so this time the result is correctly transformed into walk.
Now, let's try the word going.
So again, this remains the same because going is a noun.
Now, let's try going again, but this time or set policy to divert.
As expected, the dilemma of going is go.
Now, let's try the word Iran again, setting post a verb, as you recall, we tried this earlier with
the steamer, which just gave us back ran.
OK, so as you can see, the limits Sizer is a bit smarter and returns run.
Now, let's try the word mice, which is the plural of mouse.
Notice that we are using the stammer.
So this returns mice, which is not unexpected.
Now, let's try mice with the limited either.
So this returns, mouse, which is correct.
Note that this would be difficult to do based on spelling alone.
We need to be told that mouse is the singular form of mice.
Now, let's try the word was with the steamer.
OK, so this gets converted to NWA, which is not a real word.
It's possible that the streamer thinks this is a plural.
Now, let's try the word was with the limits of note, that was is a verb.
So this gives us back the word B, which makes sense again and notice how this is the kind of thing
that cannot be inferred by spelling alone.
We need to be given these rules in the form of a database.
Now, let's try the word is with the steamer.
So we get back is which is the same word.
Now, let's try it is with the limits isare again, a note that is is a verb.
So again, we get back the word B, which makes sense since was is just the past tense of its.
Now, let's try stemming the word better.
OK, so there is no shortage stem for this word, and we simply get back the same word.
Now, let's try to clematis better note that this is an adjective.
So this time we get back good, which makes sense, as you recall, the word better essentially means
more good.
OK, so it seems like limited vision is of limited use because we have to make sure to input the correct
parts of speech.
The question is how can we do this?
We can't manually enter the parts of speech for every word.
Instead, we would like to do this automatically.
Luckily, parts of speech tagging is another one of the fundamental NLP tasks.
And luckily, this functionality is included in NCTC.
The problem is that the parts of speech returned by the parts of speech tagger are not compatible with
the input to the limits, either.
Therefore, we'll need a function that will convert from one form to another.
So that's what this function below does, which is essentially just a bunch of if statements.
It should be pretty obvious what it's doing, but if you have any questions, please feel free to ask
them on the Q&A.
So the next step is to download the appropriate package for the parts of speech tiger.
This happens to be the average perceptron tiger.
The next step is to instantiate a sentence, I've chosen the sentence Donald Trump has a devoted following,
as you recall, we use this sentence because it's an example of where we use the word following as a
noun instead of a verb.
Note that we also split the sentence into individual words.
The next step is to run the parts of speech tagger.
Note that this returns a list containing tuples.
Each tuple contains two items which are the word in the corresponding tag.
OK, so as you can see, we get back exactly what I described.
Note that following is a noun.
The next step is to run our limits, either on each token, passing in the tax, we just got back.
Note that we convert the tax first, using the function we defined above.
OK, so we can see that two words are now different.
The word has has been limited as to have, and the word devoted has been limited is to devote note that
the word following has not been reduced.
OK, so now we're going to try our other sentence, which is the cat was following the bird as it flew
by.
Again, we'll begin by getting the parts of speech tags.
As you can see, and this example following is a verb.
The next step is to limits each word in our sentence.
So we see that in this case, following has been reduced to follow.