So in this lecture, we'll look at a more advanced method of summarizing text known as text rank.
In my opinion, this tends to work better than the simple TFI Taf method we previously discussed.
Now, because this method is quite advanced, these lectures have been split up into multiple parts.
This first theory lecture will look at intuition only, which will be for everyone.
The second lecture will look at how tech's rank really works, which is based on Markov chains and requires
a decent amount of knowledge about probability and linear algebra.
And the reason we need to do this is because there is no library to do what we're going to do.
In order to write the code, you'll have to understand what the code is doing, which means you'll have
to understand the math.
But note that I have to qualify this statement.
There is a library you can use where you simply pass in some text and get a summary.
So that will be the last example I show you in this section.
The reason that this is not the main code is because plugging your text into one line of code doesn't
really help you understand what's going on.
In fact, when it's only one line of code, it doesn't really even matter which method you use because
every method looks exactly the same.
So it's not a great learning experience in that respect.
However, if you're more application minded and you don't really have the skill or desire to learn about
how this works, you can feel free to skip to this later lecture, which I've called Text, Summarization
and Python are the easy way.
Now that being said, let's continue on to how tech's rank works.
So the best way to understand how tech's ring works is to first review how the TF IDF method works.
As you recall, this is a very simple process.
We begin by splitting the document into sentences.
We then compute ATF IDF matrix on those sentences, which gives us a matrix for each row, represents
a sentence in each column represents a word.
In other words, every sentence becomes a vector of 250 of values for each word.
We then score each sentence by taking the average of the TFR of components after we have a score for
each sentence.
The summary is simply made up of the top scoring sentences.
So where does TEX rank fit into this picture?
Well, Tex Rank is simply an alternative method of scoring each sentence instead of simply taking the
average of the TFI D of values.
We use a more advanced scoring methodology.
Importantly, note that all the other steps remain the same.
We still split the documents into sentences and we still compute the TFI Taf matrix, and we still form
the summary by taking the top scoring sentences.
The only difference is in how we compute those scores.
So how does techs rank scoring work?
Well, to understand this, you really need to understand Google's PageRank, which is what tech's rank
is based on.
As you recall, Page Rank was a state of the art ranking method for search engines 20 or so years ago,
and it's the reason why Google became one of the largest tech companies in the world.
So what is the intuition behind Page Rank?
Well, we first begin by recognizing that the internet is made up of web pages, and these web pages
are what will be potentially returned to search results.
Thus, our goal is to compute a score for each of these web pages.
Once we have a score for each web page, we can then rank the web pages by this score and then sort
our search results, according to this ranking.
So this should sound familiar.
Now the question is how do we score each web page?
So the secret to scoring each Web page is to take what is called a random walk in terms of web pages.
This simply means that for each web page, we randomly select a link on that web page.
Go to the next web page and repeat this process after doing a bunch of fancy math, which is the subject
of a more advanced lecture.
A surprising thing happens.
It turns out that the probability of landing on any single web page after a certain amount of time remains
constant.
It also turns out that it doesn't even matter which web page you started on.
That is to say, no matter where you start, if you just keep going from web page to web page randomly,
you'll eventually settle on some fixed distribution.
So if you imagine a very small internet of just a few web pages, here's what it might look like.
As you can see, the chance of landing on Page B is thirty eight point four percent for Page C. It's
thirty four point three percent and so forth.
Intuitively, you can imagine that a webpage with more links to it would be more popular and hence you'd
have a higher chance of landing on that web page.
On the other hand, web pages with only a few links or no links to it would be less popular, and the
probability of landing on those pages would be smaller.
So hopefully that makes sense.
Popular web pages have a higher chance of being landed on an unpopular web.
Pages have a smaller chance now.
What does this mean for us?
Well, these probabilities are in fact, the Page Rank scores.
Popular web pages get a higher score and less popular web pages get a lower score.
The assumption is that more authoritative, legitimate pages will have more incoming links.
Now, of course, you might wonder, well, what if people just make a bunch of websites and point all
of them to their own web site, which they want to rank highly as an exercise?
I'll let you think about why that won't work.
OK, so now that you know how PageRank works, how does this relate to text rank?
Well, the intuition is this we know that we want to score each sentence similar to how we score each
web page in PageRank.
Therefore, we treat each sentence as if it were a web page.
So in this analogy, every sentence is a web page.
The next step is to think about what's the equivalent of a link from one web page to another.
In the case of sentences, you recall that we computed the TF IDF vector for each sentence.
The number of links from one sentence to another is just the cosine similarity between their two IDF
vectors.
Intuitively, if many sentences have a high cosine similarity to one authoritative sentence, that authoritative
sentence will end up with a higher score.
But note that this is not symmetric, even though cosine similarity is.
And this is because if this one authoritative sentence has a high cosine similarity to many other sentences,
the probability of taking a random walk to any of those other sentences will be small because it has
to spread out among those sentences.
In other words, if you take a random walk among the sentences using cosine similarity as a proxy for
the number of links from one sentence to another, you will eventually have a higher chance of landing
on a more authoritative sentence.
And again, as mentioned as you take this random walk, this eventually settles on a fixed probability
distribution of sentences, and these probabilities are the scores that we will use to build our summary.
So that's the intuition behind text rank.
Now, please note that this final part of the intuition lecture is what I would consider to be intermediate
to advanced.
It does take some time to think about, so don't be afraid of it seems confusing at first.
If you found this last part difficult to understand, please skip over this for now.