So in this lecture, we will look at how to summarize text by using TFR, yes, I like this method because
it's surprisingly simple at a high level.
The way it works is this we begin by splitting the document.
We want to summarize in two sentences.
Once we have our sentences, we're going to come up with a method to score each sentence.
Once we have a score for each sentence, we're going to rank the sentences by those scores.
Our summary will simply be the top scoring sentences.
So pretty simple, in my opinion.
Also note that this is an extractive method because we extract the sentences from the given documents.
There is no need to generate text from scratch.
OK, so let's go through this process in more detail.
As mentioned, the first step is to split our document in two sentences.
Note that we call this sentence tokenization and note that there is a function for this in NCTC.
Now, once we have these sentences, we're going to treat each of them as if they were a separate document.
When we build our TFI, we have matrix.
In other words, notice how this is unlike the way we've used to fight in the past.
In previous examples, we used to taf on a set of documents so that each row was a document and each
column was a term.
In this case, since the sentences themselves are documents, well, we will get is a matrix of sentences
by terms.
Also notice that this does not require a whole data set of documents.
Since everything is computed from the sentences of a single document.
So that's one benefit of these methods.
They don't require a big data set to learn from.
OK, so once we have our TFI free of Matrix, what do we do with it?
Well, we need some way to score each sentence.
The simplest way to do that is to take the average of the non-zero values for each sentence, as you
recall.
Each sentences to free a vector is a row in order to free of matrix.
So for each sentence, which is a row, finds all the non-zero values and take their mean.
For example, if we have a row containing the values one, two and three but the rest all zeros, then
the result will just be the average of one, two and three, which is two.
Let's now think about the logic behind this process.
We know that each component of a 240 a vector tells us how often a specific word appear, but also recall
that these are weighted by the idea of term, which makes the score lower if it's a word that appears
very often across all sentences.
In other words, unimportant words have a smaller value, while more important words will have a larger
value.
Furthermore, this value will be even larger if an important word appears more often in that sentence.
Now, you might be wondering why take the mean and not the sum?
Well, as you recall, some sentences might be longer and others might be shorter since other values
are positive.
If we take the sum, then the score would be biased towards longer sentences.
By taking the mean, we are asking the model to pick the sentence that contains words with the highest
scores on average.
The second thing you might be wondering is why do we take the non-zero values and not the mean of the
whole vector?
To answer this question, we have to remember that the tier of RDF matrix is very sparse.
That is, most of the values will be zero.
The sentences with the least zeros will be the sentences with a large variety of words, but not necessarily
the most important words.
Of course, you're encouraged to test out different variations to see what works best.
OK, so now that we know how to compute the scores for each sentence, the next question to consider
is what do we do with these scores?
The answer is to sort them and then to pick the sentences with the top scores.
The question really is how do you do this?
And for this, there really are multiple options you'd have to choose which option works best for you?
I'll discuss a few possibilities, but there could be more.
If you want to think about this further.
So the simplest option would be to take the top end sentences, for example, top five, top 10 and
so forth.
Another simple option is to take the top end words or the top end characters.
This could be useful if you're building a search engine, since there is limited space for you to show
your summary.
Yet another method is to use a percentage.
For instance, take the top 10 percent of sentences or the top 10 percent of words.
Another possibility I've seen is to take all the sentences where the score is greater than some threshold.
For instance, the average score, however, that might include it too many sentences.
So you can modify that by multiplying the average score by a factor to get a new threshold.
In my opinion, this is also not ideal, since this factor is kind of arbitrary.
So try it if you like and remember to experiment to see what works best.