So in this lecture, we will be looking at a notebook to implement Tex rank in Python.
Now please note that this is an advanced notebook where we will implement the concepts we previously
discussed.
If all you want is a function that can do all this work, simply skip to the next lecture.
Now, as promised, most of this notebook is the same as the previous notebook, and the only difference
is in how we compute the scores.
Because of this will be starting this notebook at the point right after we've computed the TFI Taf matrix.
So the next step will be to compute the cosine similarity.
Now this function is very concise since we can just pass an X and get back.
Yes.
But because it's so concise, it may not be clear what it's actually doing if you've never seen it before.
Also note that s is analogous to the Google matrix, which we call G in the theory lectures.
In any case, you recall that the cosine similarity is supposed to be a function that computes the similarity
between two vectors.
But here we are, just passing in the whole Matrix X.
So what does this actually mean?
Well, firstly, note that this function comes from so I could learn if you like, please scroll up
to the top and check the import yourself.
So this function is a special function, which actually does many similarity computations all at once.
It helps to think about what we would do if this function did not exist.
Essentially, what we want to do is a double for loop.
This is because we want the cosine similarity between every sentence and every other sentence.
So suppose we had some function that could compute the similarity between two vectors.
If this is the case, then inside our loop, we would simply compute the similarity between Vector AI
and Vector J.
Furthermore, we would store this similarity in a sentence by sentence matrix called S at position i
j.
So hopefully that straightforward.
Basically, the second learn version of this function does all that work internally so that the matrix
as we get back is exactly what I just described.
Now, just as a sanity check, we're going to check the shape of s if we did this correctly.
This should be a relatively small matrix since the number of sentences is small.
Well, the number of words is large.
OK, so this result makes sense.
Firstly, note that this is a square matrix, which must be the case since its sentences by sentences.
Furthermore, this implies that we have 17 sentences.
As a further sanity check, we can also check the length of our sentences list.
And we get back 17 as expected.
Now, as you recall, our matrix must eventually be a mark of matrix, which means that each row must
come to one which is currently not the case in order for us to make this true.
We need to normalize our matrix by dividing every row by its sun.
Note that we must also pass and keep them as equals true so that the result is a 2D matrix, which ensures
that numpy broadcasting operates the way we expect.
The next step is to check the some of the first arrow of our matrix as a sanity check.
And it seems to one as expected.
Now, as you recall, this matrix is not our final matrix.
We also need to smooth this matrix in case it contains zeros.
To do this, we'll start by creating a uniform transition matrix called you.
It's simply a matrix of 1s divided by the number of sentences.
The next step, which is, again, just a sanity check, is to make sure that the some of the first
row is one.
And as expected, the some of the first row is one.
The next step is to compute our final matrix, which is a convex combination of SC and you basically
we're going to take 85 percent of this and 15 percent of you.
Of course, these proportions must sum to one so that each row in the result still sums to one.
The next step is to just double check that the first row of the result still sums to one.
And as expected, it still seems to one.
OK, so the next step is to compute the eigenvalues and eigenvectors of our matrix in this notebook.
I'm actually going to show you both methods since they both actually work and you can decide which you
think is better.
Now, as mentioned by convention, we think of the eigenvalue equation in terms of column vectors instead
of row vectors.
Because of this, Nampai implements the AISC function to work on column vectors by default, and therefore
we need to transpose our matrix mathematically.
The reason why we need to do this is easy to see if we transpose our state transition equation to be
in the standard format.
When we do this, the state distribution becomes a column vector and our matrix is transposed.
Note that this function gives us back both the eigenvalues and the eigenvectors.
So let's check the eigenvalues to make sure they are what we expect, as you recall, the theory states
that one of these eigenvalues should be one.
As you can see, the first eigenvalue is indeed one.
The next step is to check the corresponding eigenvectors.
Now this requires a bit of understanding of NumPy API.
You might be wondering why is this Colin Ma0 and not just zero by itself?
And the reason is that the eigenvalues are stored as columns and not as rows.
Therefore, if we want the corresponding eigen vector that goes with one, we need to index the first
column and not the first row.
So for now, just make note of these values.
The next step is to test our assumption that this is in fact an eigenvectors corresponding to an eigenvalue
of one, to do this, we simply multiply our eigenvectors by the matrix.
And this should return the same I can vector.
OK, so it appears that the result is the same.
So this is in fact a proper vector.
Now, at this point, if you did this exercise yourself, you may have gotten stuck.
This is because if you haven't noticed, our eigenvectors vector contains some strange values.
In particular, it doesn't look like they seem to one, nor are they even positive.
This doesn't make sense since they're supposed to represent a probability distribution.
So you might be thinking something is wrong, but in fact, this is entirely expected.
As you recall, eigenvectors are not unique.
This is because you can simply multiply both sides of the eigenvalue equation by a constant, and it
would still hold.
And of course, this constant can be absorbed into the eigen vector to make a new eigenvectors vector
parallel with the original.
So what constant should we multiply it with?
The answer is we should divide our eigenvectors by its sum as an exercise.
Please confirm that this will result in all the values being positive and all the values something to
one.
Now, before we move on, I want to show you a more direct method of finding the limiting distribution,
which might look simpler depending on how you think of the problem in this block of code.
We're simply going to find the limiting distribution by brute force.
That is to say, we're going to actually go through the transitions until the distribution converges.
To begin, we'll initialize our limiting distribution to be the uniform.
Distribution will define a threshold of 10 to the minus eight, which is what we'll use to quit our
loop.
We'll also initialize a delta variable to infinity.
Inside the loop, this will store how much the distribution has changed from one step to the next.
Finally, we'll keep track of the number of iterations of the loop.
The next step is to enter a while loop that continues.
As long as Delta is bigger than the threshold inside the loop, we increment it by one.
The next step is to compute the distribution for the next step, which we get from multiplying our distribution
by X.
We'll start this in a temporary variable called P.
The next step is to compute the next delta, which is the sum of absolute differences between the new
distribution and the old distribution.
The next step is to update the distribution to be.
When we exit our loop will print the number of iterations to see how long it took.
OK, so as you can see, this only took 41 steps.
The next step is to just print our limiting distribution.
Although you could simply check whether these numbers are the same by looking at them yourself.
Well, check this numerically later in this notebook.
The next step is to check for some of the values just as a sanity check.
As you can see, the distribution of sums to one.
As expected.
The next step is to compute the sum of absolute differences between the answer we got before and the
answer we got by brute force.
As you can see, the result is on the order of 10 to the minus eight, which makes sense because that's
what we set as a threshold.
The final step in this lecture is to set our scores to the limiting distribution, as you recall, this
distribution actually corresponds to the text rank scores, which we will then use to find the top sentences
in our document.
As before, the next step is to find the sort index for the score.
Note that since the remaining steps are the same as the previous notebook, we'll go through them quickly.
The next step is to print our summary, which is just the top five scoring sentences.
OK, so the summary is the retail sales figures are very weak.
But as Bank of England Governor Mervyn King indicated last night, you don't really get an accurate
impression of Christmas trading until about Easter.
Said Mr Shah.
A number of retailers have already reported poor figures for December.
The warnings echoed an earlier caution from Bank of England Governor Mervyn King not to read too much
into the poor December figures.
Retail sales dropped by one percent on the month in December after a 0.6 percent rise in November.
The Office of National Statistics said clothing retailers and non-specialist stores were the worst hit,
with only internet retailers showing any significant growth, according to the ONS.
So again, this seems like a pretty reasonable summary.
So as before, one thing you can do is check the summary against the title, which is like a summary
itself.
Also, you might want to check this summary against what we got in the previous lecture to see if the
results improved on other examples, I noticed a pretty big improvement.
OK, so as with the previous notebook, we're going to put all the steps into a function which will
take in some text and print out a summary.
This time we won't step through the function since it's the same as what we just went through.
The next step is to test our function on a new article, which I've chosen from the entertainment class.
So this time the summary is good REM Green Day and the Black Eyed Peas took home two awards each, as
well as Best Female.
Goodrem also took home the Pepsi viewer's choice award, whilst Green Day bagged the prize for Best
Rock video for American Idiot.
Other winners included Green Day, voted for Best Group and the Black Eyed Peas.
The Black Eyed Peas won awards for Best R&B video and Sexiest VIDEO, both for Hey Mama.
Local singer and songwriter Missy Higgins took the title of Breakthrough Artist of the Year, with Australian
Idol winner Guy Sebastian taking the honours for Best Pop VIDEO Again, this seems like a reasonable
summary.
As before, it's nice to compare our summary against the title to see how the authors summarize the
story.
And in this sense, our summary seems to work well.