In this lecture, I will be giving you an official exercise prompt in order to complete the article
spinner.
Now, technically, you already know what the exercise is.
So this lecture is more of a reminder to do it.
In addition, I'll provide a few details in case you find them helpful.
As usual, you can feel free to look at the solution at Notebook to get the data set.
But please do not go further as this will ruin the exercise for you.
OK, so the data set will be using is the BBC News data, once again, our model will be trained on
the business articles only.
So if you want to make your model like the solution, you'll need to filter out these articles.
You may find that you prefer to train on.
All the articles, or perhaps on a different dataset entirely, such as Wikipedia, whatever dataset
you use, it is up to you.
Personally, I think it would be more interesting if you used Wikipedia, since the dataset is much
larger and contains much more attacks per article.
However, it would take more work if you're interested in doing this.
You can find archived versions of Wikipedia at Dumpster Wikimedia dot org.
Note that the last time I checked these were stored as XML files, so you'll need to do some work in
order to parse these into plaintext.
Another option is to use our poetry data that is poems by either Robert Frost or Edgar Allan Poe.
I think this would be interesting as well.
OK, so once you've decided what text you want to use, your next task will be to build the model we
described.
Our model is essentially a three dimensional matrix where the two context words are used as conditioning
variables for the middle word in terms of arrays which will store in Python.
This is no different than the second order Markov model accepts.
The ordering of the dimensions will be switched around.
That is, in either of these cases.
Conceptually, you have a V by V by V matrix.
In addition, you may want to consider whether it is better to store the distributions in a Python dictionary,
as you recall, this also requires you to be able to sample from a probability distribution, which
is represented as a dictionary.
Note that we've written this code before, so there is no need for you to figure it out again.
Now, there are some details about the language model that would be worth discussing specifically,
what should we consider as tokens?
As you recall, one method is to simply drop all punctuation and use words as tokens.
Another method is to keep punctuation using nutcase case tokenize.
Another method is to simply use a string split, which keeps the punctuation by default, although in
this case, the punctuation will be stuck to the adjacent word.
In the coming solution, we'll be using an old case where tokenize, which I think is appropriate for
this task because it keeps punctuation as separate tokens.
So for example, if a sentence ends with a question mark and we replace the word before it, then the
new word will also have come from a question.
Furthermore, this will be helpful when we string the sentence back together after it has been spun.
Otherwise, it would be difficult to read the result because we wouldn't immediately know where one
sentence starts and another ends.
We also wouldn't know which sentences are statements, which are questions and so forth.
We would also lose units of measurement where there are numbers.
So for example, since we'll be looking at business articles, there will be many documents containing
monetary values.
So keeping punctuation is useful.
Now, once you have your model, you'll need to write a function to actually spin the article.
This is pretty straightforward, but still requires some work.
You'll need to figure out which words you want to replace and also how often to replace them.
For example, if you replace every word, then the result won't even resemble the original.
Nor will it be likely to even make sense.
You'll also need to consider certain details like Do you want to ever replace two words in a row?
Furthermore, you'll want to think about how to check whether a word can be replaced for some words.
There simply may not be any other options.
Since the try.
Graham is unique.
Another interesting task to consider is we'll end up with a tokenized document, but how are we going
to put it back together so that it looks like something we can read?
One way to do this is to use the joint function in Python.
However, this is not perfect because, for example, if we join each token with a space, then we'll
be putting spaces before punctuation as well.
And of course, this is not correct syntax for English.
Instead, we'll be using a class called Tree-bank Word de Tokenize or from NCTC, which can be used
to tokenized a list of tokens.
So once you've finished spinning your article, you should print the result, which displays both the
original words and the replacements so that it's easy to compare.
And this will require you to tokenize the tokens.
So good luck, and I'll see you in the next lecture.