So in this lecture, we'll be looking at the notebook for how to summarize text.
We'll begin by doing all our imports, all of which you've seen before in this course.
The next step is to download the necessary files from Ntsiki.
The next step is to download our data set, which is the BBC news data once again, as mentioned, please
feel free to try this on any document you like to test out how well it works.
The next step is to load in our data using pedigreed 6V.
The next step is to call the head to remind ourselves what is inside this data frame.
As you recall, we have text and labels.
The next step is to choose a random document note that have chosen a business document.
The next step is to define a rap function, which helps us print the results more nicely.
The next step is to print our chosen document to see what we got.
OK, so it's Christmas sales worst since 1981, some article about how sales are not good.
Note that for these articles, the title is included with the article text.
In fact, it's the first line in the text.
This is actually quite useful because the title can be thought of as a summary for the text.
Therefore, one way to check whether or not our summary is good is to see if the title gives us the
same information as our own summary.
The next step is to tokenize their document in two sentences.
Now you'll notice that this line seems a bit complex.
And this is because we're trying to remove the title from the article before we do any processing.
So to explain what this does, we first call I look zero, which gives us the document as a string.
Once we have this string, we then call the split function to split on new lines.
But we only want to split once since we only want to remove the title.
So we also pass in a one since we split it once.
This will give us back a list of two items.
The first of which is the title in the second of which is the article text.
Therefore, we choose the value at index one, which gives us back the text.
The next step is to create our team IDF vector riser object.
Note that we're using English stop words and also L1 normalization.
This ensures that we won't be biased towards longer sentences, which would have a higher score simply
due to the fact that they contain more words.
The next step is to call Fit Transform, which gives us back our TFI Taf matrix, which as usual, we
call X.
The next step is to write a function to score a sentence, given its CFR the F representation.
As you can see, it takes in a single row from the CFR The Matrix inside the function.
We start by selecting only the non-zero values.
Once we have those values, which I've called Little X, we then call the mean function.
As you recall, this is exactly what we described in the previous lecture, which was do take the average
of all the non-zero values.
In this next block of code, we're going to compute the score for each sentence and store them in a
new array called Score.
We begin by initializing an array of zeros with size equal to the number of sentences we then to afford
to live through each index from zero up to the number of sentences.
Note that at this point, we don't need the sentences themselves, but only the 240 of Matrix inside
the loop.
We index the Tier five matrix by grabbing the ith row and passing this into the function called to get
sentence score.
This will return the score for the eye of a sentence, at which point we can store it inside our scores
array.
Now, as a mini exercise, you should recognize that there is a way to make the above two blocks of
code more efficient.
In particular, it's possible to compute all the scores in one step.
So consider how you might accomplish that and then implement it yourself as a sanity check.
Compare it with the existing code to ensure that both methods give you the same result.
OK, so the next step is to sort the scores, as you recall, is not the scores themselves we really
care about, but rather they're ordering as such.
What we really need is the sort of index which we obtained by calling ARG saw it instead of just sort
of note that I've also negated the scores since by default.
It's sort of in ascending order, whereas we would like the top scoring sentences first.
The next step is to generate our summary note that have made a few comments here to explain that there
are many different options for how to choose which sentences to include.
To review, the simplest method, which is the one will use, is to simply pick the top and sentences.
Another method is to pick the top in words or even the top end characters, as you recall.
This can be useful for applications such as search engines, where the space to include your summary
is limited.
Another option is to use a percentage instead of just the number, and the final options are to pick
the sentences above some threshold, like the average score or some multiple of the average score.
Note that I've made one final remark here, which is that using the methods where we pick the sentences
above some threshold, it's not necessary to sort by score.
Instead, we might wish to keep the sentences in order.
This could make a lot of sense.
For example, if you're reading a story, you don't want to read the end of the story before the beginning
of the story.
If it's a news article, that could also make sense since news articles are written such that the most
general facts show up first and the details show up later.
So these comments here are probably more important to consider compared to the actual code itself.
In any case, the code is as follows we simply loop through the first five elements of the sword index
inside the loop.
We then print the score along with the corresponding sentence.
OK, so the summary is as follows A number of retailers have already reported poor figures for December.
However, reports from some high street retailers highlight the weakness of the sector.
The UN has revised the annual 2004 rate of growth, down from the 5.9 percent estimated in November
to three point two percent.
Our view is the Bank of England will keep its powder dry and wait to see the big picture in a British
Retail Consortium survey found that Christmas 2004 was the worst in 10 years.
OK, so not a bad summary.
Now, as you recall, I mentioned that one way to check whether or not this is a good summary is to
compare it with the title, which is itself kind of like a summary.
Note that this is the same quote as before, except that we take the first element instead of the second,
which gives us the title.
In fact, we've already seen this title, which is Christmas sales worst since 1981.
Now, at this point, we should realize that whether or not our summary is good is really a subjective
concept.
It's really up to you whether or not the summary suits your needs.
There isn't really a concept of accuracy, since it's possible that many different summaries would be
sufficient.
There are summaries which would be obviously bad, but again, that's more of a subject of assessment.
Finally, note that objective metrics for summaries do exist, such as the Rewe score, but understanding
that requires even further study, which is outside the scope of this cause.
Furthermore, it's still not perfect, so the benefits you're learning it at this point does not outweigh
the cost.
Now you recognize that the above process took quite a bit of code, and this is the kind of thing that
we would like to put inside a function.
So the next step is to implement a function that performs the steps we just went through.
This function will take in a document and print the corresponding summary.
Since you've seen all these steps before, we'll just go through them quickly.
So our input variable is called text.
Inside the function, we tokenize the text into sentences.
We then conferred the sentences into a two year free of Matrix.
The next step is to score each sentence using the Get Senate score function we define above.
Once we have the scores, we then sort the scores and get back to sort index.
Once we have the sword index, we then loop through the first five values and inside the loop, we prints
out the score along with the corresponding sentence.
OK, so let's test out our new function on a random document.
This time I've chosen a document from the entertainment class.
OK, so the summary is the Black Eyed Peas won awards for best R&B video and Sexiest video, both for
Hey Mama.
The ceremony was held at the Luna Park fairground in Sydney Harbour and was hosted by the Osborne Family
Good Round, Green Day and the Black Eyed Peas took home two awards each other.
Winners included Green Day, voted Best Group and the Black Eyed Peas.
The VH one first music award went to share, honouring her achievements within the music industry.
OK, so that seems like a pretty good summary.
Now, let's prints out the title of this document, which is kind of like a reference summary for us
to compare to.
OK, so the title is go to RIM wins, top female MTV prize.
Now, in my opinion, our summary is better because it tells us what multiple bands have won the title
only tells us about one particular artist.
And it's not clear why this is the most relevant fact, such that it would make up the title.
The downside to our summary is that it doesn't tell us what awards ceremony the article is about, which
is something you would probably expect of a summary.
In fact, because of how news articles are written and even simpler way to summarize them would be to
take the title in the first few lines.
So let's see what we get when we print the whole article.
As you can see, the first sentence does, in fact, tell us what awards ceremony it's about, which
is the Australian MTV Music Awards.
So again, practically speaking, if it's a news article, remember that they are always written from
general to specific.
So it may be simpler just to take the first few lines in terms of machine learning.
Recall that this is an example of applying domain knowledge since we know how news articles are designed.
We can use that information.