So in this lecture, we will be looking at the code to implement our second order language model to
generate Robert Frost poems.
We'll start by importing Nampai and String.
We'll also set the NumPy random seed so that you get results which are consistent with this video.
Note that this isn't required and you are encouraged to turn this off to see other examples.
OK, so the next step will be to create dictionaries to represent our probabilities.
The first dictionary, called Initial, will store the initial state distribution.
The second dictionary, called First Order, will store the First Order transition probabilities, but
only for the second word in each sentence.
This is unlike the First Order Markov model, where the matrix was counted for all one step transitions.
Of course, you could count the words that way, too.
It's simply that I've decided not to.
The third dictionary, which I've called Second Order, will store the second order transition probabilities.
As you might expect.
The next step is to define a function call to remove punctuation, which will remove punctuation from
each line of text.
This is optional, so if you'd like to leave us out, feel free to do so as before.
For this course, there's no need to worry about what this code actually does, since it's not machine
learning.
The next step is to download our data set, which is a text file containing Robert Frost poems.
The next step is to define a function called add to.
So this function will take in three items a dictionary, a key and a value.
The basic idea is the key will represent some starting word or pairs of words in the second order case.
Recall that Python dictionaries can be any type of object.
The value for this dictionary will be a list storing all the possible next words.
So as a quick example, suppose that we're modeling Second Order transitions from a text corpus.
We come across the phrases I am happy, I am sad, I am content, I am happy, I am happy, I am hungry.
In this case, the key to the dictionary would be a tuple containing the tokens I and -- the value
would be a list containing happy three times and the rest of the words.
Once the goal is, we're going to eventually turn this into a dictionary where the word points to the
probability of that word appearing.
But the first step is to simply collect the words from the text.
The next step is to traverse our data set where we make use of the function we just wrote to populate
the dictionaries above.
So we'll start by looping over each line of the Robert Frost text file inside the loop.
The first step will be to our strip and lowercase.
The line will then remove the punctuation and then call the split function to tokenize the sentence.
So the result of this will be a list of tokens.
The next step will be to grab the length of our list of tokens, which we will call big tea.
The next step is to look through each index from zero up to Big T inside the loop or grab the IV token,
which we'll call T.
The next step will be to check whether or not I is equal to zero, if it is, then this is the first
word in a sentence.
So we would like to update our initial state distribution.
We can do this by incrementing the value stored for T by one note that we use the get method.
So if T does not exist in the dictionary yet, it automatically gets a count of zero as before.
The first step is only to gather the counts after we've gone through the whole data set.
We can normalize this distribution.
The next step is to look at the Ellis block, as you may have inferred, the L'ESPOIR takes care of
any token when I is bigger than zero.
In other words, this is not the first word in the sentence.
Therefore, the first thing we can do is grab the previous word in the sentence at Index II minus one,
we'll call the results underscore one.
The next step is to check whether or not we are at the end of a sentence, if we are, then I is equal
to Big T minus one.
In this case, we would like to add a special second order transition by essentially creating a fake
token.
As you can see, this fake token is called end in all uppercase letters.
The purpose of this is when we are generating new poems, we know that the line should end if we sample
the end a token.
And of course, this is required.
Otherwise our line would go on forever.
So pay attention to the arguments we pass into the add to debt function.
The dictionary we want to update is the one for second order transitions.
The key for this dictionary is a tuple containing T underscore one and T.
The value for this dictionary is our fake and token.
The next step is to enter a new if statement where we check if I is equal to one.
Note that this is a separate if statement since we might want to run both statements on the same token.
In other words, they are not mutually exclusive as an exercise.
Think of a situation where both statements would be run.
OK, so what should we do if I is equal to one?
This is when we are looking at the second word in a sentence.
Therefore, we would like to update the First Order dictionary note again the arguments into the adjudicate
function.
The key is the previous word, which is to underscore one.
The value is the current word, which is T.
The next step is to look at the Ellis block.
Basically, this looks at any case we have not yet considered, which is when I is not zero and I is
not one that is to say we are not looking at the first or second word.
In this case, we should have at least two previous words to look at.
So we can grab the second last token at Index II minus to the next step is to update our second order
dictionary again.
Notice the arguments.
The key is the previous two words and the value is the current word.
Now, one thing to keep in mind is that at this point, the initial state distribution is the only dictionary
that contains actual counts.
Therefore, it's the only dictionary we can normalize directly for the other dictionaries, they simply
store samples of possible next words.
We have yet to turn those into counts.
So in this step, we're going to normalize the initial state distribution first.
We'll start by grabbing the sum of all the values, which gives us the total.
Of course, this is just equal to the number of lines we encountered.
The next step is to live through each key value pair in the dictionary for each pair.
We're going to take the count, which I've called see and divide that by the total.
As you know, this is our maximum likelihood estimate for the probability of each token.
OK, so the next step is a bit more difficult, as you know, for the transitions, we don't yet have
counts.
We simply have lists which store all the possible next words.
So the following function called list to predict does two things.
The first thing it does is it creates a dictionary of counts.
The second thing it does is once it has the counts, it then normalizes the counts, which converts
them into probabilities.
OK, so the input argument to this function is T-S, which is just a list of tokens inside the function.
We'll start by creating a new empty dictionary called D.
The next step is to grab the total number of samples, which we'll call end.
This is just the length of time.
The next step is to live through each token into yes.
Inside the loop, we simply increment the value for the token by one each time we encounter the token,
as before we use the get method so that we can say the default value is zero.
Once this loop is done, we will have a dictionary starring the counts.
So the key is the token and the value is the corresponding counts.
The next step is to then loop through each key value pair in the dictionary inside this loop.
We simply divide the count by the total, which gives us the probability for each token.
Once we're done, we return.
OK, so the next step is to make use of our new function.
We'll start by looping through the first start a dictionary which stores the probability for each second
word in a sentence.
As you recall, what we previously stored in this dictionary was a list of possible next tokens.
We now call the list to repeat its function to simply replace that list of tokens with a dictionary
of probabilities as an exercise.
You should think about what structure this dictionary will have.
Once we are done, basically convince yourself that the result will be a dictionary of dictionaries.
The next step is to use our function again, but this time on the Second Order dictionary, the main
difference with this dictionary is that the key is now a tuple of two words instead of just one word.
But we're still doing the same thing as before, which is converting a list of words into a dictionary
of probabilities of words.