So in this lecture, we will be looking at the notebook to implement the articles, spinner and code.
We'll begin by downloading our data set, which is the BBC news dataset.
As we've seen in previous lectures.
The next step is to do our imports of note, here is the text wrap module.
This is helpful when we print out our spun article.
Since if we don't wrap the text, each paragraph will go off the screen.
Also make note of the tree-bank word.
The tokenize are class, which is used to tokenize a list of tokens back into a single string.
In the real world, this is necessary, since that's how your article will eventually be presented.
The next step is to download the necessary data files for NCTC.
The next step is to load in our data using PD that reads the we.
The next step is to call the DFAT head to remind ourselves what our data looks like.
So as you recall, we have two columns, which are the text and the labels.
The next step is to cast our labels to a set to remind ourselves what labels we have.
As you recall, we're going to be choosing just one of these labels to train our model.
As mentioned, you can feel free to use the whole data set if you like, or even a completely different
data set, which you find interesting.
The next step is to set our label of choice, which for the purpose of this lecture will be business.
Please feel free to pick a different label if you like.
The next step is to grab only the text for only the rose that match our chosen label, as always.
Recall that you can read this from the inside out.
The inside part is where we select only the rose that match our label.
Once we've done that, we can grab the appropriate column, which is text.
Once we have the result, we'll call the head function once again to check the result.
OK, so as you can see, the result appears to be a panda series containing only business articles,
which is what we expect.
The next step is to create our model, as you recall, we can break this into two parts.
The first part is to count the number of possible outcomes.
And the second part is to normalize the counts so that they become probabilities.
So this block of code pertains to the first part.
We'll begin by creating an empty dictionary called Proverbs.
Note that this will be a dictionary of dictionaries.
The key for this dictionary will be a tuple of context words.
In our case, that's the previous word and the next word.
The value for this dictionary will be another dictionary for the nested dictionary.
The keys will be possible middle words.
The value will be the corresponding count for the corresponding middle word.
The next step is to look through all of our documents inside the loop.
We will first split the document into lines.
As you recall, each document contains multiple paragraphs, which are separate lines.
This includes the title of the article and the text.
The next step is to live through each line or, in other words, loop through each paragraph.
Inside this inner loop, we will then tokenized the line by calling word tokenize.
This will give us a list of tokens.
The next step is to loop through the tokens.
Now you'll notice that I'm not leaping through the tokens directly.
This is because on each iteration, we'll need to grab three tokens simultaneously.
That is to say, we are actually iterating through tri grams.
This is why we want to loop through the index instead of the token and also why we stop it then tokens
minus two.
So inside this third, we start by grabbing our three tokens notice that these are three consecutive
tokens that index I I Plus and I plus two.
The next step is to form the key to our dictionary, which is a tuple containing T. zero, A. two.
Obviously, T one is the middle word.
The next step is to check whether or not this key exists in our probs dictionary.
If it does not, we should create an entry for it, which will be initialized as an empty dictionary.
The next step is to increment the count for the middle word to you want to do this?
We have to handle two cases the case where two you one is not present already and the case where it
is.
So if Taiwan is not present, we simply set the count to one.
However, if Taiwan is president, then we increment the count by using plus equals one.
OK, so hopefully you're convinced that by the end of this loop, our dictionary will contain at the
proper counts.
The next step is to normalize the counts we just collected to do this.
We're going to begin by looping through each key value pair in Proverbs.
In this case, he represents the tuple of context words and represents the probability for the middle
word.
Except these are not probabilities just yet.
For now, they are just counts in order to turn them into probabilities.
We'll need the total count, which is the sum of all the values in D.
Once we have the total, we can then loop through the itself.
In this case, the key is the middle word and the value is the count inside this loop.
We turn this into a probability by dividing the current count, by the total count.
The next step is to print out our problems dictionary to confirm that it has the correct format.
OK, so it appears that our dictionary looks like it should.
The key is a tuple of the two context words and the value is a probability dictionary.
Note that for some programs, there is only one possible middle word for these cases.
We probably wouldn't bother to try and replace those words because the result would be no change.
However, note that for some keys, there are many possible words.
For example, make note of the key US giants.
In this case, many programs could make sense to US agrochemical giant U.S. banking giant, US foods
giant, US media giant, and so forth.
So typical business news.
Also notice how our punctuation is tokenized, so you could have jumped one point eight percent or jumped
ten point seven percent.
OK, so at this point, our model is complete.
The next part is to figure out how to actually use the model to spin articles.
Conceptually, it's simple, but as you'll see, there are a few details we need to contend with.
Firstly, let's recall that each line in a document is a paragraph.
So if we grab the first article at I look zero and we split on a new lines, we should be able to see
a list of paragraphs.
So notice that some lines are simply empty, which is because our articles are formatted to have an
empty line between paragraphs.
The next step is to write a function called spin documents.
The main idea of this function is that it's a higher level function.
Basically, we're going to break our problem down into slightly smaller parts, which is to spend each
paragraph one by one.
So the point of this function is to simply call another function, which will spin each paragraph.
It's also responsible for joining the paragraphs back together at the end so that the result has the
same format as the input.
OK, so as input, we take in a variable called Doc, which is a string representing the entire document.
Inside the function, we begin by splitting the document by new line, as you recall, we just did this
above, so you know what the result looks like.
We'll also create an empty list called output, where we will story each spun paragraph as they are
received.
The next step is to enter a loop, which will loop through each line or each paragraph inside the loop.
We will check whether the line is empty or not.
If it is not, then we call the spin line function, which we have not yet seen.
The result is called New Line.
Otherwise, the line is simply an empty line and we can set New Line equal to line.
Once we have a new line variable, we can then append it to our output list.
Once we finish a loop, we can then join the output by a new line character, which will reverse the
split we did above.
So the main thing on your mind at this point should be what does the spin line function look like?
But before we get into that, we need to think about how to tokenize a list of tokens.
As you recall, we need to tokenized each paragraph in order to spin the article.
But once it's been spun, we need to turn the list of tokens back into a string.
The hard part about this is sometimes tokens should be joined by a space, but sometimes they should
be joined without a space.
For example, if we have two words side by side, we would like to join those with a space.
But if we have a word and punctuation, then we would not want to put a space between them in order
to do this.
We're going to make use of a class called tree-bank word de tokenize or will begin by creating an instance
of this class.
The next step is to pick a random sentence from one of our documents.
At this point, we just want to print this out to confirm that if we tokenize and then de tokenize this
sentence, we end up with the same sentence.
So now that we know what our sentence looks like, the next step will be to call word tokenize on this
sentence.
This will return a list of tokens on that list of tokens.
We are then going to call tokenized using our tokenized objects that we just created.
OK, so as you can see, the tokenized string is the same as the original string.
This means that punctuation has been treated correctly without spaces at the appropriate points.
The next step is to define our sample word function, which will sample a random word from a probability
distribution represented as a dictionary, as you recall.
This is the same as what we've seen before, so I won't explain it again.
The next step is to implement our spin line function as input.
This takes in one line as a string inside the function.
We begin by calling word tokenize to get a list of tokens.
We'll also initialize an index to start at zero, which will be used while we live through the tokens.
The reason we are not using a for loop will become clear later on.
We'll also initialize their output at this point, which will be a list containing just the first token.
Note that the first token cannot be spun because there is no previous word.
The next step will be to answer a while, loop up to lend tokens minus two as before.
The reason we stop at minus two is because we'll be indexing three tokens at a time.
Inside the loop will grab a three token set indices eye plus one and eye plus two.
The next step will be to form our key, which is a tuple containing the previous tokens is zero, and
the next tokens to the next step is to grab the probability distribution corresponding to this key.
That is the distribution of all possible metal.
We'll call the result p dist.
The next step is to check whether or not we should replace this metal work.
So the next if statement defines when we should do this note that we are using two criteria, the first
criteria is obvious and that is that the length of penis must be bigger than one.
If it is not, then there is no possibility of replacing this word with something else.
The second criterion is stochastic.
We generate a random number and then check whether or not that random number is below zero point three.
Basically, this will control how often words are replaced.
As you recall, the random function draws the number uniformly between zero and one.
Therefore, by using a threshold of zero point three, we're saying there's a probability of 30 percent
to replace each word that can be replaced.
OK, so if we want to replace this middle word, then we answer this if statement, the next step is
to find a replacement for the work.
To do this, we simply call the function sample word passing in the probability dictionary p dist,
we'll call the result middle.
Now, because it would be nice to compare each replacement with the original, we're still going to
a penalty one to the output list.
The next step will be to upend middle surrounded by angle brackets.
Since these brackets are unlikely to occur in the news articles, it should be unambiguous when we see
these in the results.
Finally, we'll also append the next word to what this means is that we will never replace two words
in a row.
It's not that you can't, but I've simply decided not to.
In this case, because we've effectively added two new words to the output, we're going to increment
I by two.
So this is the reason we did not use a for loop.
If we could use default, we would always be incrementing by one.
But this lets us control how much to increment I.
The next step is to look at the Else. block, in this case, we are not replacing the metal work.
Therefore, we simply append a T one to the output list.
We also increment I by one instead of two.
OK, so by this point, we will be outside the wire loop.
There is one final thing we need to consider, which is the value of AI at this point.
I could be either one or two steps away from the end of the sentence because we may have incremented
AI by one or two on the last step of the loop.
Therefore, the behavior will be different depending on what we just did.
Now, if we increment it AI by two, that means we replace the second last token and we appended the
final token to the output list.
And since we increment it, I buy two.
That means I will be greater than LEN tokens minus two.
In that case, there is nothing else to do in the other case where we did not increment I by two.
That means we did not replace the second last token.
In that case, we increment it.
I buy one.
And so I is equal to LEN tokens minus two.
In this case, we still have yet to append the final token.
So that is what we do at this point.
The final step, once the output list is complete, is to call the tokenized function to convert our
list of tokens back into a single string.