So in this lecture, I'm going to walk you through the exercise for classifying the SMS spam data set
and a few extra things that I think will help you understand the data.
So hopefully you had a chance to try this out before watching this lecture.
So the first thing we're going to look at is our list of imports.
Note that most of this should be familiar.
Some notable additions are the Rowsey, a U.S. score, F1 score and the confusion matrix.
In addition, we'll also be using the word cloud library, which will let us visualize the most common
words from each class.
This will be helpful in understanding why our model behaves the way it does.
In other words, why it thinks an email is spam or not spam.
The next step is to download the data.
Next, I'm going to load in the data as usual, since we have a CSB.
We're going to use the pandas reads as we function.
Also notice I'm passing in a special encoding here called the ISO eight eight five nine Dash one.
Knowing what this is is an important, but the reason why we need it is because the CSB contains some
invalid characters.
So if you tried to use the default encoding, you're going to get an error that there's a character
that's not valid in UTF eight.
Generally, when you're working with text, you're going to come across invalid characters, especially
these days with the existence of emojis and other non-standard symbols.
OK, so the next step is to do what you have done, head to see what our data frame looks like.
So as you may have expected, we have two columns, one for the labels and one for the text.
Now, for some reason after you load, in this case, we are going to see some extra columns that are
just empty.
So what we're going to do is clean up this data frame a little bit by dropping these columns.
So here we have some code to drop the unwanted columns.
The next step is to do what you have done had again to see what the result is.
OK, so now you can see that our unwanted columns are gone.
The next step is to rename the columns we kept, which are currently just V1 and v2, which are not
really descriptive names, so we're going to rename them to labels and data.
The next step is to do it had again to ensure that our data frame is in the format we want it to be
in.
OK, so now our data frame looks a lot better than it was before.
The next step is to draw a histogram of our labels.
This will help us determine whether or not we have imbalanced classes.
OK, so as you can see, in this case, we do have imbalanced classes in particular.
Ham is much more common than spam, so it will make sense to look at other metrics such as the F1 score
in the AUC.
So the next step is to create a new column called B Labels, which assigns a value of zero to ham in
a value of one to spam.
We're also going to extract this column as an umpire, Ray.
Note that you probably don't need to do this since last I remember.
So I could learn accepts data frames and series as input arguments, and it allows your labels to be
strings.
But if you were to write your own model, you'd want the labels to be represented numerically.
So in general, it's good practice to do this.
The next step is to call train, to split, to split up our data into train and test.
Note that you could even do cross validation so you can calculate on average how well the model does
on the validation set.
That's probably a more accurate measure than just one train to split.
OK, so the next step is to create our X matrix, which contains the input features for every sample
for this experiment.
I'm going to use the count vector riser from Saikia Learn, which just gives you the raw counts.
I have code here for TF IDF that's commented out.
In case you want to try that to notice how I'm passing in the decode error argument sets you ignore.
This is because, as I mentioned earlier, if any invalid UTF eight characters are found, we want to
just ignore them.
The next step is to just print out X train to see what we got back.
OK, so we get back a sparse matrix instead of the usual Nampai array, where you can see all the numbers,
as you recall.
This is because the array contains a lot of zeros.
And so this is the most efficient format to store the data in.
OK, so now that we have our data training, the model in evaluating the model is super simple.
We just call the objects constructor, then we call the fit function and then we call the score function
for both training and test.
OK, so we do pretty well, about 98 percent accuracy on the test set.
One interesting thing you can try later is to use TFI Taf or use a different classifier and see if the
results improve.
OK, so recall that our classes are imbalanced and thus accuracy may not be the best measure of performance.
It could be the case that our model is just predicting ham all the time.
Since that is the over-represented class, in order to make sure that this is not the case, we should
check other metrics.
The first alternative metric we're going to try is the F1 score.
To do this, we're first going to get our model's predictions for both train and test, which we'll
call P train and P test.
Note that we've assigned these two variables since they will be used again later in this notebook.
The next step is to call the F1 score function, passing in the targets and the corresponding predictions
for both train and test.
OK, so as you can see, the F1 scores in the 90s for both train and test, which is a good sign that
our model is performing well for both classes, it's another way of saying that both precision and recall
are good since, as you recall, the F1 score is the harmonic mean of these two.
In the next block, we're also going to check the AUC as yet another alternative measure of performance,
as you recall.
This function requires our model's posterior probabilities, so we need to call the predict proper function.
In addition, we want the columns at index one.
Since these represent the probabilities for class one will assign these to the variable names prob train
and probe test.
The next step is to call the RLC a U.S. score function, passing in the targets along with the corresponding
mortal probabilities for both train and test.
So as you can see, our model does very well on this measure, obtaining close to one for both training
test.
The next step is to see if we can get a more fine grained view of a model's performance by looking at
the confusion matrix.
Note that we don't have to compute these ourselves since there's a function inside it.
Learn that can do this for us.
However, if you did want to compute this yourself, it would be quite simple.
So please try that as an exercise if you would like.
OK, so as you can see, this only returns an array, which is not that useful to look at.
You can probably guess based on these numbers what each entry means, but that would be an ideal.
Instead, a better method is to plot the confusion matrix.
Now, normally you can do this in cycle learn, and that is the case right now.
However, you'll notice this comment I made here, which explains the situation basically at the time
I've made this lecture circuit learn version one is out.
But as is typical with these data science Python libraries, these updates come with breaking changes.
Unfortunately, the confusion matrix is one of those breaking changes as the function to plot the confusion
matrix has been moved.
Unfortunately, this new version of cycle learning is not currently easy to install in CoLab, so we
won't be looking at the new way to do this in this lecture.
Now, breaking changes tend to always confuse a lot of beginners, so it's also not good to use the
current version either.
The solution for this notebook is to simply implement a confusion matrix plot ourselves.
With the help of seabourne and pandas as an exercise, I would recommend looking up the new way to do
this, and I can learn as a version one and to try that out on your own machine.
In any case, you can see that we've defined a function called plot scheme, which takes in a confusion
matrix as input.
Inside the function, we define a list of classes which are just ham and spam.
The next step is to convert the confusion matrix into a pan as data frame, which is useful because
both the columns and the rows can have names.
In particular, we are going to name them by their class names, which will show up on our plot.
The next step is to call the seaborne heat map function with several arguments to make it show annotations
and to format the numbers correctly.
The final step in this function is to label the axes, so we know which one corresponds to the prediction
and which one corresponds to the target.
After defining the plot, the function, we're going to call the plot system function on our train confusion
matrix.
OK, so as you can see, it looks about right on the diagonal, which represents correct predictions.
We have the largest numbers on the off diagonals.
We have relatively small numbers.
Note that this goes for both classes, so we rarely predict hammer spam and we rarely predict spam as
ham.
The next step is to plot the confusion matrix for the test set as well.
So again, we see the same pattern where most of the predictions are correct for both classes.
This makes sense in light of our high F1 scores and AEW sees.
So in this example, because we have access to the raw data, there's some interesting stuff we can
do.
So I've created this function called Visualize.
What this is going to do is create a word cloud.
Basically, that's a picture where more frequent words appear larger and less frequent.
Words appear smaller.
So for each of the classes, we'll be able to see what are the most common words in a spam message?
What are the most common words and a ham message?
So this function takes in a label which can be either hammered spam.
We then grab only the rows that contain that label, and we loop through the data column.
We keep a running string of each message we encounter and just concatenate them together.
This is because that's what the word cloud library expects to get its input.
So once we create the word cloud, we then use map plot to show it.
So let's run this block.
OK, so let's first call this on the spam data.
OK, so we see that some of the common spam words are text call now free mobile call, text and so forth.
So that makes a lot of sense.
So now let's call our function on the ham data.
So here we see that the common words for him, Hammer, love will.
OK, now go and so forth.
So it's a lot different than the spam messages.
OK, so in this next block of code, we're going to do a simple analysis to figure out what our model
is getting wrong.
There shouldn't be too many examples since we're getting about 98 99 percent accuracy.
So we create a new column called Predictions, and we set it to the predictions generated by our trained
model.
The next step is to create a variable called sneaky spam.
It's sneaky spam because it's able to bypass our spam filter.
So to do this, I need to filter the data frame by selecting any rows where the prediction is zero,
but the true label is one.
So we use that element wise and operation.
So if you don't know how to do this now, you know, then we loop through the data field and print each
message.
So here are some sneaky spam messages.
OK, so we have free message, Hey there, darling.
Did you hear about the new divorce Barbie?
Do you realize that in about 40 years?
OK, so you can see from this that some of these are obviously spam and some of them are not quite obviously
spam.
So overall, it's a mixture of the two.
OK, so in the next block of code, we're going to create another variable called not actually spam.
Since these are messages that are spam classifier to Texas spam, but are actually legitimate messages.
So it's the same process as before, except I've switched the zero in the one.
So here it's not actually spam messages.
OK, so what's very interesting about this is that apparently these are not spam, but if you read these
messages like, Hey, great deal farm tour, blah blah blah.
OK, so for a lot of these, you can actually understand why these might be considered spam.
It actually makes a lot of sense why these would be misclassified.
So, for example, unlimited texts, limited minutes.
So that could be something that was actually spam or something that your friend is sending you to give
you some information.
You also have stuff like this at the bottom where you know, we have send aid for customer service.
So that actually looks like a spam message.
So perhaps some of the roads are mislabeled.