Everyone and welcome back to this class natural language processing in Python.
In this lecture, we will continue looking at our code to implement a cipher, a decryption algorithm.
This lecture will focus on creating the language model and the associated functions.
Let's start by defining the data structures we need.
First, we need a marker matrix to store all the bigram probabilities.
As stated earlier, since there are 26 starting letters and 26 ending letters, we must have a 26 by
26 matrix.
We'll call this em.
We also need a vector to store all the unit grant probabilities, but to be more specific.
This will represent the initial state distribution.
In other words, we don't care about unit grams that appear later in a word, or we just care about
the first letter.
So we'll call this PI and it has 26 entries, one for each letter.
Next, we need some functions to help us actually populate these data structures pie in M..
First of all, have a function to update them and we'll call it update transition.
It takes in two arguments.
One need to see each one represents the starting character and the two represents the ending character.
The important thing to remember is that in order to index a matrix, we must use integers since the
number of rows and columns of The Matrix are both 26.
Then the indices will be from zero to 25 inclusive.
Luckily, we have a useful python function that can convert characters into integers.
It's called ORD.
Just as a side note, if you are not a computer scientist, you may have never seen anything like this
before, so to describe what's happening?
Remember that everything stored in your computer is stored in binary code, just zeros and ones.
A series of zeros and ones is called a base to no system, just like how the system, as humans use
to count, is a base 10 number system.
In any case, the important thing is that everything stored inside a computer is represented by a number.
Those numbers happen to be zeros and ones, but they are still just numbers at the end of the day.
So you have zero zero zero zero zero zero zero one zero zero one zero zero zero one one and so on.
Now, ASCII characters, because they are also just things stored in a computer are also represented
by numbers.
It just so happens that this function AUD takes in a character and returns its integer representation.
In other words, the number that it's represented by.
So a long time ago, some computer scientists decided which characters should be represented by which
numbers.
So a lowercase a is 97, a lowercase B is 98, lowercase C is 99, and so on.
Luckily, these are all in order, so there's no need to worry about the letters not being contiguous.
In other words, all we need to do in order to get integers that are appropriate for indexing our matrix
is just call the old function and then subtract 97 by doing this.
A will get mapped to zero, B will get back to one and so on up to Z, which gets mapped at 25.
And remember, that's because all the letters are contiguous.
Finally, once I have I and J, then we add a one to M at position RJ.
Now you might be wondering if all we're doing is counting, then these are not proper probabilities.
We'll deal with that later.
Also, notice that I've initialized the Matrix M to start with ones and zeros.
This is because we want to use add one smoothing as described in the lectures.
Next, we have a function to update PI, which uses the same principle we take in one argument, which
represents the first character in a word.
And then we map that word to an integer called I.
Then we index PI by I and add one again.
It's not a probability yet, but we'll deal with that later.
Next, we have a function to get the log probability of a single word, the function we look at after
this one will help us get the log probability of an entire sentence that should be easy to write.
Once we have this function, so this function takes in one argument, which is a single word inside
the function.
We get the unique grand probability by taking the first letter in the word word at index zero.
We convert that into an index and then we find its probability by indexing PI.
Next, we take the log since we would prefer to work in log space and assign this to a variable called
log p.
Next, we loop through the rest of the characters in the world.
Notice how we index the word variable with one and a colon.
This means started index one and go all the way to the end.
Inside the loop, we convert the current character C-H into an integer j using the same method as before.
Notice that the variable AI still represents the previous letter, so now we can index em at position
I.J to get the next bigram probability.
We take the log of this and then add it to log P.
Notice how we're using the plus equals operator here, so it just adds whatever is on the right to the
existing value.
Next, we have to remember to update EI, which is supposed to represent the previous letter to J.
The current letter.
This is because the current letter becomes the previous letter on the next iteration of this loop.
Once we're done looping through all the letters, we found the log probability of the given word, so
we return the log P..
Finally, we have a function called Get Sequence Prop, which returns the log probability of an entire
sequence as input.
It takes an argument called words, which can be one of two things.
First, it can be a string containing multiple words i.e. a sentence, or it can be a list containing
multiple words, each stored as a string.
What we'll do is if the input argument is a string, then we'll call words that split in order to convert
it into a list of strings.
So after this first block of code, we can be sure that the words variable is a list containing strings,
each of which are individual words.
Next, we initialize a variable called Log P two zero.
Then we loop through each word in our list of words, inside the loop.
We call the previous function get word probab to get the probability of the current word.
Then we add the result to log P. And we keep doing this until we've seen all the words.
Once we're done, we have the log probability of the entire sentence.