Hey, everyone, and welcome back to this class natural language processing in Python.
In this lecture, we are going to continue looking at our cipher, a decryption script, the previous
lecture looked at creating some functions that would help us to build our language model and then use
it to evaluate the likelihood of a word or sentence.
This lecture will focus on training the language model.
This will involve working with actual data.
So the first thing we need to do is download that data.
So in this first block of code, we first check to see if we have a file called the Moby Dick Act text.
If we don't, then we'll download it using the request library.
Notice that the file is just stored on my website.
So if you're having any problems downloading this file, please let me know.
So we call requests that get and this story is the response in a variable called R. At this point,
the response is not yet stored in a file.
So next we open a new file called Moby Dick in a write mode, and then we write the contents of the
response to this file.
So we call it right.
And as input, we pass in our content to decode.
We need to call the decode function because our content is a binary and not a string.
So the decode function turns it into a string.
Next, we're going to populate our bigram and MoneyGram data structures.
First, we need a regex that will help us remove any non alphabet characters.
That's anything that's not a letter.
We'll call that variable regiment's.
Next, we enter a loop that reads the Moby Dick file.
Line by line inside the loop.
The first thing we do is strip out any whitespace by calling our strip.
Next, we say if line.
The reason for this is some lines in the file are just blank since we called the strip function previously.
Then if this line is blink, it'll now be an empty string, which would have failed this condition.
So we only want to answer this if statement if there are actually words in this line.
Next, we call Regex Dot Sub and replace any non alpha characters in a line with an empty space.
Technically, you wouldn't have to do this if you wanted to also encode and decode it, not alpha characters
or otherwise write code a deal with them.
But let's keep this simple.
Next, we lowercase all the letters in the line.
And then we call the split function to get individual words.
As per conventional A.P. terminology, I'm going to call these tokens next, we live through each token
a one by one.
Inside this loop, we're going to call the two functions we defined earlier for updating in them.
Remember that these only update the counts first, we grab the first letter token at Index zero.
We pass this into the update PI function.
Next, we lived through all the other letters starting from the second letter.
So that's why we index token by one and then a call in.
Inside this loop, we call a function update transition, passing in zero and one.
We also have to remember to update C zero to be the current character C one.
Finally, when we're finished looping through each line of the book, there is one more thing we need
to do.
Remember that Pi and them currently only contain counts.
They are not yet true probabilities.
In order to make them into true probabilities, Pi Masoom to one in each row of them must sum to one.
We can accomplish that simply by dividing pie, by its sum and dividing each row of them by its Roseanne.
So that's why these two lines do.
If you want, I would encourage you to do a simpler example to prove to yourself that this code actually
does what we think it does.
Alternatively, another option you could take here is to compute the log right now, if you recall our
other functions, which calculate the log likelihoods are currently responsible for taking the log.
However, that's inefficient.
There's no need to take the log of the same elements again and again.
Instead, you could just store the log probabilities directly in variables.
So you might want to call them, say, log PI and log in and then use those directly in the other functions.