In this lecture, we are going to continue our discussion about how to convert text into vectors using
counting.
At this point, we understand the basics of tokenizing a string into a list of words and then counting
each word and putting those counts into a vector.
But there are still a few details we need to explore.
In this lecture, we are going to consider the concept of stop words.
So suppose that we did our word counting procedure as described.
What do we think might happen?
Well, let's consider some very common words, words that appear everywhere, but do not necessarily
carry a lot of information.
Words like and the it is and so forth.
What might happen if we counted these words?
One thing to consider is whether or not these words are useful.
Pretty much any text can contain these words, no matter what topic they are about whether or not they
are spam, whether or not they are positive or negative.
So it could be said that these words don't carry useful information for whatever task we are doing.
Here's another thing to consider.
As you recall, increasing dimensionality is generally bad.
This will be a common theme throughout this course.
So don't worry if you don't understand everything there is to know about this right away.
But just keep in mind that we prefer not to have very high dimensional vectors.
One simple reason is that the larger they are, the more computation we have to do.
More computation takes more time and more space.
Here's another thing to consider putting high dimensionality aside.
What about the fact that these words do not help us differentiate between different kinds of documents?
Recall that one method we use to gain an understanding about vectors is their distance to each other.
But if we make a vector that contains lots of ends, lots of verbs and lots of it, all the documents
will look like similar vectors.
The counts for these words will overshadow the counts for the other, possibly more important words
like mitochondria or voltage.
So it may be useful to simply ignore these words altogether.
We call this list of words that we want to ignore.
Stop words.
Luckily people have already thought about this issue and thus libraries like Psyche Learn allow you
to remove stop words from the counting process automatically.
So when you create your account event to raise your object, there is an argument called stop words
that allows you to specify the stop words by default.
This is set to none.
And so if you don't specify any stop words, then they will be capped.
There are two other possible options.
The first is to pass in the string English.
If you do this, there is a built in list of stop words in English that will be ignored.
This is probably the easiest thing to do, but unfortunately it only covers English.
The second option is to pass in a list of user defined stop words.
So perhaps you're working in a different language, or you're working in a specific domain where the
stop words are not the same as they are for generic English documents.
For example, you might be working in a business or an industry that uses very niche terms, but some
of them are not important since they are used to commonly.
In this case, you can specify your own stop word list.
Now, if you want to do an LP in another language, one possibility is to use stop words from.
Okay.
So here is some sample code to show you how to get a list of stop words in other languages.
The first step is to import and I'll take the next step, if you haven't already done so, is to call
notes.
Download Passing in the argument stop words.
This will download the stop words for each language.
The next step is to import the Stop Words module, and at this point you can call the words function
passing in your language of choice.
So for example, you can pass in English, German, Spanish and even Arabic.
But note that not all languages are supported.
To get a full list of languages, check the directory shown on this slide.
If the language you want is not there, you may still be able to find a list of stop words simply by
doing a search.