In this lecture, we will be looking at the code to implement Markov models for building a text classifier.
As always, I strongly encourage you to treat this as an exercise and to try to code this yourself from
scratch.
You already know what to do and what the results should be.
So if you'd like to treat this as an exercise, please look only at the first block of code where we
download the data sets.
The only thing left to do is to translate our ideas into Python, which does not require any new concepts
or any new knowledge.
So please try to complete this by yourself first.
As output, you should display the accuracy of your text classifier on both the train and test set.
OK, so assuming you've already attempted the exercise, let's continue as mentioned, we'll begin by
downloading the data center, which consists of poems by Edgar Allan Poe and Robert Frost.
Our goal will be to build a classifier that can distinguish between the two authors using one line of
a poem, so each line of a poem will be a sample.
Now, just as a reminder for beginners, please do not try to type these URLs by hand.
Simply copy and paste the links in the notebook itself.
The next step is to import everything we need for the rest of the notebook.
The next step is due to find a list that contains the file names for our text files.
The next step is to run the head command to see what's inside our tax files.
We'll start with the Edgar Allan Poe.
OK, so as you can see, this file contains poems by Edgar Allan Poe.
Note that some lines are empty.
Also note that there is capitalization and punctuation.
The next step is to run the head command for Robert Frost.
Again, we see that some lines are empty and that there is capitalization and punctuation.
The next step is to collect our data into lists for this experiment.
We'll be treating each line as an input sample.
This is opposed to an entire verse or an entire poem, so we'll begin by creating two empty lists,
one to hold on to the text and one to hold on to the labels.
The next step is to loop through each input file notice that we use enumerate, which gives us the index
at the same time.
So the first of all, we'll have the index zero and the second file we'll have the next one will be
using these as labels for our data set inside the loop.
We're going to print out the file name along with the corresponding label.
This is so that we'll know which label corresponds to which author.
The next step is to live through each line of the file inside the centre loop.
We begin by calling our strip, as you recall, when you look through a file line by line in Python.
It includes the new line character at the end.
The airstrip call will remove this new line since it's not needed.
The next step is to call the lower method, which converts all the text to lowercase.
The next step is to call if line.
As you recall, some lines are empty, so the following code will only run if the line is not empty.
OK, so inside the if statement, we begin by removing punctuation.
Note that understanding this code is not crucial at this point.
You can easily just copy and paste this code from Stack Overflow.
So after this line, our text will be lowercase and all punctuation will be removed.
The final step is to append this text to our list of input texts and to append the label to our list
of labels.
OK, so as you can see, Edgar Allan Poe has been assigned label zero and Robert Frost has been assigned
Label one.
The next step is to do a train test split.
Part of the challenge in this notebook is figuring out when to do your train to split.
It might seem natural to do this later after we've converted the text into integers and so forth.
But this way is more sensible.
As you recall, the test set should emulate texts that we have never seen before, so it may contain
unknown words that don't match up to any integer index.
The next step is to check the length of Y train and Y test to see how many samples we have.
The next step is to print out a few rows from the train text.
As you can see, these are random lines of palms, and everything is lowercase without punctuation.
The next step is to print out a few rows from Y Train.
As you can see, we get a mixture of zeros and ones since the train to split function also shuffles
the data set.
The next step will be to convert our text into integers, as you recall.
These will be indexes into the markup matrices we are about to define.
We'll begin by setting ADX to one which will act as our current index as we loop through the text.
We'll also initialize a word to index dictionary with one entry, which maps the unknown token to zero.
Basically, this will never be used for the train set, but it may be used for the test.
If the test set contains words that do not appear in the train set.
The next step is to loop through our train text in order to populate where 280X inside the loop will
begin by splitting the text into tokens, as you recall.
There are a few options for tokenization, but calling the split function is the simplest thing to do.
The next step is to leave it there.
Each token we find inside the loop, we start by checking whether or not the token is already in our
word to ADX dictionary.
If it is, then there is nothing to do if it is not.
Then we need to create a new entry for this token.
So inside the if statement, we start by assigning the current value of index for the current token,
the next step is to increment IDEX, so it will have the right value for the next token we encounter.
The next step is to simply print out word to RDX.
Feel free to check this yourself, to see what words it contains.
The next step is to check the length of where to ADX.
Note that this will determine the size of our markup matrix and the initial state distribution.
The next step is to convert this data into integer format.
Recall that this is what we need to index our in our pie.
So the first step will be to create two empty lists, which we will use to store the results.
The next step will be to loop through the train tracks inside the loop will begin by again splitting
the text to do tokenisation.
The next step will be to use a list comprehension to map each token to its corresponding text.
The result of this will be a list of integers representing one line of a poem.
The next step is to append this list of integers to our list to find above.
The next step is to do the same procedure for the test set.
This is mostly the same as the above loop, but there is one major difference, as you recall.
It's possible that not every word in the test set appears in the trains there, so we can't naively
try to index the word to dictionary.
Instead, we'll use the get function and will ensure that we always return a default value of zero.
As you recall, this corresponds to the unknown token.
The next step is to print out a few rows from the list, which is populated.
So as you can see, the result is a list of lists of integers as expected.
OK, so at this point, we're ready to build our A and pie matrices which represent the mark model.
Now one important thing to remember is that we don't just have one Markov model.
We'll have as many as there are classes.
Since we have two classes, we'll have two A's and two pies.
The first step is to assign the length of word to IPX to a variable called V.
By convention, this is what we normally use to represent the vocabulary size.
In the theory lectures, we often use the letter M A for generic states.
But since we know we're working with language, it's OK to use V.
The next step is to initialize A0, Pi zero A1 and PI one.
You'll notice that I've initialized each of these arrays to all ones.
The reason for this is we're going to be using add one smoothing.
So these ones are the initial fake counts for each initial word and each transition.
The next step will be to add the rest of the counts from the actual text.