So in this lecture, we will be looking at the notebook where we build TF IDF completely from scratch
for this notebook, once we build our TFI, the effectors, what we will do is pick random documents
from our data set and look at which words end up having the largest TFT of components.
What we expect to see is that words with large TFI of values should display two traits.
Firstly, they should be words that appear often in the documents.
That means their term frequency is high.
Secondly, they should be relatively unique compared to the whole data set.
That means their document frequency is low and hence their inverse document frequency is high.
So this is our hypothesis about what should happen before looking at the results.
So we'll begin this notebook by downloading our BBC news data once again.
Note that this is an arbitrary choice.
You should feel free to choose any data set you like.
The next step is to do our imports.
Note that for this script, our inputs are quite simple because we'll be implementing TF IDF using only
basic components.
The next step is to download the necessary data for MLC, where tokenize are.
The next step is to read, in our case, we're using PD that reads, Yes, we.
The next step is to call DfT head to remind ourselves what our data looks like, although this step
seems to be quite trivial.
I always find that printing things out helps me through the coding process.
The next step is to populate our word to a next mapping, as you recall.
This is necessary so that we know which word maps to which columns in our RDF matrix will begin by initializing
index to zero.
And we're 280X to be an empty dit.
We'll also create an empty list where we will store our tokenized documents.
This is helpful since we'll need these tokens later.
Why don't we do our accounts?
The next step is to loop through the text column of our data frame inside this loop.
We begin by lower casing and tokenizing the current document we call the result words.
The next step is to create an empty list called Duck.
Is it since we will be building our word to index dictionary, there is no need to store the words themselves.
Instead, it would be more efficient to simply store the indices that they map to.
The next step is, do you live through each word inside this loop?
We check whether or not the current word is in word to RDX.
If it is not, then that means this word it doesn't yet have a corresponding index.
So if this is the case, we add this word to our dictionary where the key is the word in the value is
the index.
The next step is to increment index by one so that it has a new value the next time you use it.
At this point, we know that our word has a corresponding index because we either just created it or
it already existed.
Therefore, we can grab its index from word to ADX and then store it inside our list.
Doc as it once we're outside the inner loop, our isn't list is complete or in other words, the sentence
has been converted to integers and then we can store that inside our tokenized docs list.
OK, so at this point, we're to index mapping is complete, and we have converted all of our documents
into lists of integer IDs.
The next step is to create the reverse mapping as well.
So this code iterates through our word to index dictionary and simply makes the value the key and the
key, the value.
Note that this is somewhat inefficient because we're using a dictionary data structure, even though
our indices are only integers from zero up to the vocabulary size.
It would be more efficient to store a list instead.
Since the index to a list is already an integer as an exercise, you may want to think about how do
you store it that way instead?
The next step is to find the number of documents which we will call end.
Note that this is simply the number of rows in the F.
The next step is to find the number of unique words which we will call.
We note that this is simply the size of words.
Why acts?
The next step is to instantiate our term frequency matrix.
Note that this is the same thing as our count vector matrix.
Now, in practice, although it would be more efficient to use a sparse matrix that makes things a bit
more complicated without improving your understanding.
Thus, in this script, we will be using a dense matrix, which is just a regular Nampai array.
Using a sparse matrix will be an exercise to extend what you've learned in this lecture.
OK, so as you can see, we initialize this matrix to be a matrix of zeros of size n by V.
The next step is to populate our term frequency matrix, to do this, we simply loop through our tokenized
documents.
As you recall, we already have them all stored as lists of integer IDs.
Note that we use Enumerate, which gives us the index for the document at the same time, which we will
denote with the letter I.
So it tells us which document we are looking at inside the first.
We have a second loop which goes through each word index in the documents.
Since these are already word indices, we'll call each of them J.
Thus, to populate our term frequency matrix.
We simply increment the value at position I.J by one.
This means that in document I, we saw the J ith word one more time.
The next step is to compute the IDF term.
Note that the IDF term can be derived from the term frequency matrix.
So let's think about how that can be done.
Remember that we will have an IDF value for each word.
Thus, we will have v IDF values or, in other words, a vector of size v.
Thus, what we need to look at are the columns of our term frequency matrix.
One word at a time.
So for each column, the term frequency matrix contains a zero.
If the word did not appear in a specific document and the number greater than zero if it did.
Thus, what we are interested in is when TAF is greater than zero.
By doing this Boolean operation, we get back a Boolean matrix of the same size.
As you recall, a Boolean value is either true or false.
But in Python, it true is treated like one and false is treated like zero.
Therefore, if you want to know how many values are true, we simply need to sum up all the values.
So that's why we call the same function.
But note that if we call the some function with no arguments, it will just sum up the whole matrix
and give us a single number.
And that number will be the number of ones in the whole matrix.
But this is not what we want.
Instead, we want to have the sum over each column or equivalently the sum over each word.
Therefore, we must specify access equals zero.
We call the result document free.
So this is a V size array, which contains the number of documents that each word appears in.
Of course, this is not the IDF vector just yet.
As you recall, the IDF value is n divided by the document frequency and then logged to squash down
the value.
The next step is to complete our computation of the TFI Taf.
Now you might think that this would not work, since t.f is an end by matrix and the IDF is just a v
size vector.
However, recall that NumPy automatically broadcasts when you try to perform operations on objects of
different dimensions.
As an exercise, you might want to try this the slow way to verify that Nampai will do this broadcast
incorrectly.
In fact, you really should do this exercise because it will help you think through how we compute the
TFI t.f using t.f and IDF.
The next step is to set a random scene so that we get consistent results for the next portion of the
notebook.
You can feel free to turn this off if you want to see other examples.
So in the next block of code, what we are going to do is choose random documents from our dataset and
prints out the top five TF IDF terms.
As you recall, our goal is to basically make sure that these makes sense in terms of what we expect
to find.
You have to do so to quickly explain this code.
We start by choosing a random integer from zero up to when this will be the document index.
The next step is to index ADF at this index.
This will give us back one row of ADF.
The next step is to print the label for this row.
This will help us further confirm that our results make sense.
Well, want to make sure that the resulting words correspond to the label?
The next step is to print out the first line of text.
As you recall, each news article contains multiple lines.
However, if we print out the whole article, it will make things more difficult to see.
So to do this, we simply split the text on the new line character and grab the zero with return value.
The next step is to print out the top five terms as determined by their TF IDF value.
We'll begin by grabbing the TF IDF vector at the iith row of our ATF IDF matrix.
We'll call the results scores.
The next step is to sort the scores.
Now there are two things to know.
Firstly, when we call sort this by default sorts in ascending order.
Thus, if we want the top score to come first, then we should negate the scores.
Secondly, it's not the actual score we care about, but the ordering of the score.
Thus, we call our exhort instead of sort, which gives us back the indices in the sorted order.
The next step is to loop through the first five indices inside this loop.
We print out the word corresponding to the index.
So here you can see why we needed the reverse mapping to go from index back to word.
Otherwise, we would have just printed the index and it wouldn't be clear which word it belongs to.
OK, so let's try our function.
OK, so our first two randomly selected article is about sports.
The headline is Athens memories soar above.
Those in the top five terms are Paula Athens 1500 Meter, Her and Kelly.
So from my perspective, all of these makes sense, except for the word her, which seems more like
a stop word.
Perhaps it's the case that this word simply doesn't appear to often in many news documents, and thus
the IDF value is high.
However, the other terms, such as Paula Athens, 500 Meter and Kelly makes sense.
These are likely to be unique to this article while also appearing frequently in the article.
So let's look at another example.
So this article is about politics.
The headline is Clark faces ID cards, rebellion.
So these terms also make sense.
Cards, Clark Rebellion, ID and Bill.
In some sense, you can think of these words as the most important words in the article or the words
that best represent the article.
Notice how these are all terms which are relevant to this article and again, are likely to appear often
in this article, but probably not all articles.
So let's look at another example.
So the next example is another article about sports again, we see terms which look like they would
be unique to this article.
Note that Hingis is a tennis player and we see terms like 30th and 95th, which makes sense because
in tennis, we often care about player rankings.
OK, so now that you understand the basics of how to build a fighter, yes, here are some ways you
could extend this exercise.
The first exercise is simple.
As you recall, one part of the script involves creating the count vectors.
Of course, we already know of some code that can do this, which also returns a sparse matrix.
This is the count victimizer inside can learn.
So this exercise involves replacing our counting code with the count vector riser instead.
This should be relatively simple, but recall that the count victimizer also does another step, which
is to create the word to index mapping.
So you'll have to check the documentation to find out where this is.
The second exercise is more difficult for this exercise, you should use spies see us, our matrix objects
instead of a dense Nampai array.
Note that this is not just a matter of replacing and zeros with the CSR Matrix constructor.
The reason for this is our existing code is not compatible with CSR Matrix.
In particular, the part where we increment the value at position I.J is not allowed or it may be allowed,
but it is very inefficient.
So you should figure out the other ways you can initialize a CSR matrix and use one of those instead.