So in this lecture, we you're going to discuss how to improve the count riser using a method known
as T of IDF.
So DFAT is a very popular method in NLP specifically for document retrieval and text mining will begin
this lecture by discussing the motivation for why we might need to fight yes and the drawbacks of the
victimizer that it solves.
Well, then look at how T IDF works so that you understand why it does what we say it should and how
you might coded yourself.
OK, so let's start from the beginning, which is to answer the question what is wrong with the victimizer?
Let's consider why would you stop words as you recall?
Stop words are used as a list of words to filter out.
These are words we do not want to keep.
Now, let's recall why we do not want to keep these words.
These are words like the it and so forth.
So why do we not want to count these words?
The answer is that these words are very unlikely to be helpful for any NLP task we want to do if we
want to do document retrieval or build a search engine.
These won't be useful.
Every document probably contains the words of the IT and so forth.
Doing a search based on these words is likely not a good idea.
What about spam detection or sentiment analysis?
Again, these words are unlikely to be useful for either of these tasks.
Both spam and non spam emails will contain words like these.
Sentences with positive and negative sentiment will both contain words like these.
Now, here's a question to consider how do we know that our list of stop words is correct?
The answer is we do not.
In fact, stop words can be application specific.
For example, the word mitochondria might be useful if we want to differentiate biology documents from
documents about physics.
But what if all our documents are about biology and they all contain the word mitochondria?
In this particular case, that word is not so useful, so perhaps it would be useful if we could automatically
identify good words and bad words in terms of what will help us model our documents.
That is, we would like to be able to follow some general principle instead of having to manually curate
a stop word list.
Let's also recall why we like to have stop words in terms of the resulting document vectors, as you
recall, our method of converting text into vectors is by counting the value for each component in our
vector is the count of how many times each word appears in our document.
Stop words tend to appear a lot.
They are like glue words that helps you form a coherent sentence.
Thus, their accounts tend to be very large, and large components in a vector will completely overpower
the other smaller components.
For example, if we're trying to find clusters of vectors or we're trying to find the distance between
vectors, those large components will have more influence than the smaller components.
And we don't really want words like the to have such a large influence.
So the basis for a TFI Taf is essentially what we have already discussed.
That we want to ignore our words that appear in many different documents.
That's why they don't help us when we want to differentiate between different documents.
In other words, when a word appears in many different documents, we want to scale down the vector
component for that work.
So, for example, the word verb may have a high count due to appearing many times in a document.
What we would like to do with TFI Taf is to scale this count down based on some count of how many documents
it appears in.
So the complete solution for this issue is the TF IDF method TFI IDF stands for a term frequency inverse
document frequency.
Intuitively, it does what I've just described.
Now this is not the final form just yet, but I want to give you the intuition first.
So in the numerator, we have the term frequency.
This is simply the count that is we count how many times the word appears like we normally do.
But instead of just the count, we also divide by the document frequency.
That's what we mean when we say the inverse document frequency, and this value is based on how many
documents this word appears in.
And of course, if the word appears in many documents, then this value will be larger, which will
make the tough idea of smaller.
OK, so I hope you get the idea on the top.
We have the count which increases when the word appears more often in a single document on the bottom,
we have the document frequency, which increases when the word appears in more documents.
But since it's on the bottom, it decreases the value of the component in our feature vector.
OK, so now that we understand the intuition, let's look at the TF IDF in full.
We'll start with the most common variation, but later in this lecture, we'll discuss other variations
as well.
So let's see if IDF is actually the multiplication of two terms the term frequency and the inverse document
frequency.
Well, discuss each of these terms one by one.
The first thing to notice is the arguments of each term notice that the term frequency has two arguments
T to use the term and this is the document.
This makes sense because each term will have a different count for each document.
For example, document one might contain the word mitochondria five times, while Document two might
contain the word gravity zero times.
However, the idea of term only has one argument, which is the term T.
This makes sense because the IDF isn't specific to any particular documents, only to a particular term.
The IDF is measured by summing over all documents.
For example, we can say the word mitochondria appears in 10 out of 100 documents in our corpus.
OK, so let's start by looking at the term frequency components.
As mentioned, this component is simply the count, which is the same thing as the count vector riser.
In other words, when you call the transform function with Saikia learns count of exerciser, this t.f
matrix is what you get back.
Also note that it should be obvious why this is a matrix.
There are two arguments corresponding to the rows and columns.
It's also helpful to think of the size of this matrix.
Suppose that we have unique terms and unique documents.
In this case, our t.f matrix would be of size and by V or V by end, depending on how you orient the
matrix by convention.
Although we write it with T first and then D, we typically store this matrix as an end by V matrix.
Of course, this makes sense, since this is the same shape as the output of the count vector riser.
The next step is to consider the IDF term.
Now, earlier, I said that for intuition only you can think of this as one over the document frequency.
However, that's not precisely correct.
What do we actually want to do is represent this as a proportion, and we take the log of this value.
So suppose that NFT is the number of documents that contain the term tea as before and by itself is
the total number of documents.
Therefore, and of T divided by N is the proportion of documents that contain the term T.
We then take that inverse of this proportion and then we take the log.
This is the IDF.
So the only curious part about this is why we take the law.
The first thing to note is that it doesn't break any rules.
Since the log is a monotonically increasing function, if an over NFT is larger than the log of that
will also be larger.
Thus, the TFI ETF will go down as the term appears and more and more documents.
The main idea is that the log squashes down large values.
So imagine that the term T appears in just one document, but we have one million documents.
Without the log term, we would end up multiplying by one million.
With the law term, we scale this down to about thirteen point eight, which is a much more sensible
value in terms of feature vectors for machine learning.
Note that there are deeper reasons for taking the log as well.
This involves information theory, which is outside the scope of this course.
But please check extra reading dot to see if you would like to learn more.
OK, so now that you understand how to fight works, let's discuss how one might apply it in Python
as before.
We'll discuss how to do this in psyche learn.
In fact, you'll notice that this is very easy because there is a class called TF IDF Vector isAre,
which has essentially the same API as the count vector riser.
Therefore, the steps remain the same.
We begin by creating an instance of the TF IDF Vector Riser class.
The next step is to fit the victimizer to the training set.
We can also transform this into a TFA IDF matrix in a single step by calling Fit Transform.
The next step is to transform the test set by calling Transform Note again, that we do not want to
fit to the test set because our vocabulary and IDF values should come from the train set.
Note that like the count of exerciser, the TSA I.D. affects, Riser also has many options you can choose
in the constructor.
So, for example, you can use stop provides around tokenize, air strip accents and so forth.
So at this point in the lecture, we understand how to free of works and what issue it solves.
We know that the TF IDF is made up of two terms the term frequency and the inverse document frequency.
The next step in this lecture is to discuss some variations on these items.
Note that none of these variations are necessarily better than the others.
As always, if you want to know whether or not something will work on your data set, the best and only
option is to try it on your data set.
So let's look at some variations for the term frequency.
One method is to simply represent them using binary values zero or one, one of the word appears in
the document and zero if it does not.
In fact, this method can be used in the count victimizer as well.
Another method, which is sometimes presented as the default for the term frequency, is to normalize
the count by dividing by the sum over all terms in the document.
As you recall, the result is that each term now represents the proportion of time that the word appears
in the document.
This can also be thought of as a type of normalization.
Yet another method is to take the log of one plus the count.
This has the effect of reducing the influence of extreme values.
Again, recall that the log function squashes down large values.
The larger they are, the more they get squashed.
OK, so now let's look at some variations of the idea of.
One alternative is to use the smooth ADF in this case, we add one to the denominator, and we also
had one after taking the law.
Basically, this prevents us from ever getting a value of zero, which is possible if NFC is equal to
N.
In that case, we get log of one, which is zero.
This prevents us from possibly dividing by zero.
Later in the code, another option is the IDF Max.
In this case, instead of using end, we use the maximum term count from the same document and thus
the ratio inside the log is relative to the current document instead of the whole data set.
Yet another variation is the probabilistic IDF in this case we use and the mindset of T instead of just
n note that this quantity gives us the log odds, also known as the large it.
One final variation on the IDF I want to discuss is the trivial IDF, where we just said it's a one.
In this case, the IDF just becomes the term frequency itself.
So one final variation on the TFI ETF I want to discuss is normalizing the entire TFI ETF vector, as
you recall, we can do the same thing for account vectors.
So to remind you what we do is we take the existing ETF IDF vector and then we divide it by its L2 norm.
This makes it so that the length of every TFI together vector is one.
As you recall, one advantage of this method is that it makes ranking by Euclidean distance or cosine
distance yield the same results.
Now, unlike the count vector riser, this type of normalization is implemented and so I could learn.
So if you'd like to use it, you simply set the norm argument in the TFI IDF constructor.
This takes in two possible values, which can be either one or two.
Note that L2 is the default, so if you don't pass in anything, your TFI IDF vectors will all be normalized
to have unit length.