So in this lecture, we're going to discuss the next exercise for this course, which is to implement
the process we just described for how to summarize text.
This lecture will recap the steps you need to complete.
For the data, say please feel free to use any data set you like, will be using the BBC News data once
again, which you can get the euro for from the upcoming notebook.
OK, so the steps are as follows The first step, of course, is to pick an article you would like to
summarize.
I recommend trying this on multiple articles to see how well it works for different examples.
Once you have your article, you want to split it into sentences, as you recall.
This can be done using the tokenize function from Nozick.
Once you have your sentences, you can then compute the t.f IDF matrix passing in these sentences as
input.
This will give you a sparse matrix where the number of rows is the number of sentences and the number
of columns is the number of terms.
Once you have your 240 of Matrix, you can then score each sentence by taking the average of the non-zero
values from each row in this matrix.
The next step is to sort these scores in descending order.
You want to make note of this ordering, since that will be the ordering you use to build your summary.
In particular, your summary will simply consists of the top scoring sentences.
As mentioned, you can feel free to use any method you like for choosing how many sentences to keep.
So that sums up the exercise.
Good luck, and I'll see you in the next lecture.