OK, so in this lecture, we will be looking at how to apply in grand models to the article spinning
task.
This will be similar to the markup models we studied in the past, but with a twist.
Let's begin by reviewing what do we know about Markov models so far?
Suppose that we like to build a first order Markov model for language.
That is, we'd like to be able to predict the next word from the current word.
In this case, we can estimate a probability distribution of next words, given current words.
As you recall, this is simply done by counting.
If we want to know how often what follows T minus one, we count, how many times that occurs and divide
that by how many times w t minus one appeared as a unit gram.
But suppose we'd like to have a richer model.
Suppose that instead of depending on just one password, we would like to depend on two passwords.
In this case, we would build a second order Markov model for language.
We want to know the probability that what appears given that W T minus one end w t minus two already
appeared.
Again, we estimate this by counting the number of programs for T minus two W T minus one and T in our
text corpus.
We then divide this by the bigram count four W T minus two and W T minus one.
So one way we saw that we could use the trigger model was to generate poetry that is given the previous
two generated words randomly select the next word and so on and so forth.
Four articles spinning our problem is a bit different.
It's not that we want to generate texts from start to end.
Instead, our job is to replace text.
We want to replace text in such a way that it still makes sense in the context of the text that came
before, as well as the text that comes after.
So here is a simple idea.
How about we create a distribution for W.T. given w t minus one of the previous word NWT plus one the
next word?
This is still a trigger model, except that the dependency structure is a bit different.
Now, luckily, the way that we estimate this program probability is pretty much the same as what you
would expect.
It's simply that count of the program T minus one T NWT plus one divided by the count for the context
window w t minus one NWT plus one.
OK, so hopefully you're convinced that this is pretty much exactly the same approach that we used before.
As you recall, this is the maximum likelihood estimate.
Now, it's probably a good idea to make sure that this approach even makes sense.
Consider the context, words production and to hear some actual middle words from the dataset will be
using.
So some examples are began, capacity closer continued and facilities.
You can imagine how all of these could be used in a sentence.
Production began to speed up in Q1.
Apple's production capacity to build iPhones increased.
The CEO brought production closer to the company's headquarters and so on and so forth.
Note that there is one potential issue that may arise.
Not all of these metal words are the same part of speech.
Sometimes it's a noun, whereas other times it's a verb, whereas other times it's an adjective.
So it's very possible you'll encounter a weird looking result if you replace a verb with a noun or some
other part of speech that does not belong.
In this case, you are changing the grammar of the sentence, which will not work.
There are a couple of additions you could use to try and fix this issue, which are exercises at the
end of the coming notebook.