Hey, so let us use support to make the machines and try to implement the spam classifier.
So for this particular classification, I have imported the spam data.
So it has a data of multiple values like these are the messages.
So these messages contain different spam with data and different good important messages also.
And it has one more variable, which is the target.
And the target even consists of either the Moelis spam or is it.
So we will discuss about different libraries, which we have important, we will explore the data set,
we will look at the distribution of spam and not spam data, then we would apply text analytics and
feature engineering.
Finally, we will apply predictive analysis.
For this particular date.
So these are the libraries which I have imported, the libraries are no Findus, my.
Then I have imported council from collections.
I have important feature extraction model selection, knife based metrics and SVM from the Skillern
Library.
And I have important image from if I can display.
I have also imported warnings so that I can prepare the warnings and ignore the warnings, which we
all which I created during this run of the.
And this is the magic statement which allows the plot to be generated inside the Jupiter book itself.
So the first step is to import the data.
So here we are importing the spambot CSB this data consists of different messages.
So we will, first of all, look at the distribution of the data.
So here you can see that we have taken out the value count of the first column.
So the value comes around five thousand and around eight hundred, you can see out of five thousand.
There are eight hundred spam messages.
So we have given the account plus blog and we have plotted this a bar chart on top of it.
And we have provided the colors of the bar chart, which you want to block that is blue and orange.
And we have given the title for the.
Here is the bar chart that we have.
Now, next, us, we will plot the future of this particular data.
So, again, we have taken the same data that is found and we have plotted it.
So here the plot contains eighty seven percent messages and 13 percent spam messages.
Now, we would apply text analytics on top of this data, so we want to find the frequencies of the
words in the spam and non spam messages, so the words of the messages will be the modern feature.
So we will be converting these messages into different count vectors.
And these count vectors will have frequencies of each and every word, which we will be getting after
filtering out the stop words or the punctuations or after applying limited ization or stimming.
So whatever treatment we want to apply on top of this data, we can apply that treatment and retrieve
all the data to we those inventers which you want and then create the tokens out of.
So here we are using the counter.
And we are joining the data from the column one, wherever the data value is, wherever the column one
value is.
We are getting the data from it.
And then joining those in count one.
Then we are creating a dictionary or datastream out of.
Next, from the state of Maine, we are renaming the columns as wards in non spam and the count of them.
Next, we are creating two of the veto with the value is spam.
Wherever the data is of spam, we are getting the data and finding out the most common 20 votes from
the most common 20 words.
We are again creating a data frame and via the.
Columns for Divya created these two data friends, frenzied, every one of them do with 20 most common
words from the spam data.
I have printed a different one and the same for you, so here you can see that the words in non spam
are to you.
I though and I in you is my need for that.
So these are different words which are most common in the non spam data while in the spam the most common
ones are.
Do you call Paul?
So these are the words of a journalist from.
Now you can see.
That the most common words, which we have to have the words which are actually Stopford, so this means
that it is very important to remove these words from these dataset so that we can actually get the words
which are impacting the spam and ham classification.
So let us plot the words like which words most common, so here we have the plot for frame one.
So these are the words which are most common and these are the words which are most common in the spam.
Now, you can see that in case of spam, mostly someone is being asked for the calling, so let us see
how this actually impact.
Now we can see that the majority of frequent wards in both classes are stop words.
So stop for to be referred to the most common words in the language.
There is no simple universal list of support.
So we will try to create a list of stop words and to remove those.
So the first thing which we do is.
We removed the stopwatch to improve the analytics because most of the words forward, so they will always
end up with what we are trying to do.
So we are using future destruction, dot, dot com victimiser and from the Gulf mycorrhiza, we are
getting the Stopford sequel to English, which means that in the count is it we want to remove the stopwatch.
So these are the words that you want to remove.
So now we are fighting and transforming this particular data that this video I'm getting the X, so
X has a shape of fifty five seventy two eight four zero four, which means that we have created more
than eighty four hundred new features like this.
Eighty four hundred votes.
The new feature in the ruly is equal to one if the word W appears in the text example, it is Zettl.
If not so, if a particular word is present in the text, then a particular word will be present in
the future.
If the word is not present in the text, then it will not be present.
So.
Here you can see that X is a sparse matrix, so that means that it saves space wherever required.
Next, we will apply predictive analysis.
So our goal is to predict if one of us is spam or not.
So we assume that it is much worse, classifying non spam, then misclassifying spam.
So if there is a message which is not spam and we classify it as spam, then it would be a more difficult
situation because in that case we might miss out on the important message.
So that is why we will give priority to a non spam spam message in comparison to spam message.
So now we will transform the video this fall, as I'm not spamming to buying any value, so we give
spam to be one and have to be zettl.
So we will simply apply a map and whatever values in the column, one would represent it this time it
will change it to one, and if it is, it will change.
And then we are simply applying this split on top of it.
So here you can see the shapes so extreme has a shape.
Thirty seven three three eight four zero four.
The shape of excessed is adding three eight three nine.
And it was beautiful.
Now we will apply the multinomial name base and we will be using the regular recession barometer Alpha.
So we will evaluate the accuracy, recall and precision for the modest.
So.
We have got the score three, that is we are putting in zero values for the score brings for different
scoring values and four alpha in the list of Alphas that does values from one one hundred thousand.
We are generating values, Skippy.
Community values.
So the bees via is the object which we are creating from the name based model that is multinomial name
based and they are applying bees to it.
Now, we would apply school train, we are getting out of school every so busy in school, get us extra
and why don't we just score using the Bayesian method, then we are scoring.
The testing data.
Then we are putting the metrics that this recall score to the recall test and the precision school to
the precision best.
This is what each and every value now we will see the Force10 learning models, which are being created
for different alpha values.
So these are different values for which we have created the models.
And you can see these other inaccuracies and best accuracies which we have to treat.
So we select the model with the most, best precision.
OK, so we will select the model, which has the highest, best precision.
So the best model comes out to be the one with the alpha value, fifteen point seven three and the three
accuracy is a zero point nine seven.
The best accuracy is zero point nine six.
The best goal is zero point seven seven, and the best precision is one.
Now, here we have used this particular model because it does not produce any false positives.
So this is what we are focusing on because we don't want any spams.
No one seems to be going into the spam folder.
So we are focusing on hundred percent precision.
So we are getting the best precision from the top models.
So these are the different best precision, these are the best precision.
So we will select some model which has the maximum accuracy out of these.
So the one which comes out with the maximum best accuracy is we are selecting it using ide x max.
So it is the model with alpha value, fifteen point seven three zero one zero.
So this is the model which actually has the maximum, best precision and corresponding maximum best
accuracy.
Now, next, we will generate the confusion matrix of this so we can generate the confusion matrix using
matrix dot confusion matrix, and we can simply apply the same color method which we have.
That is, the columns will be predicted zero and predicted one, and the index would be actual Zyda,
an actual one.
So these are the values which we get.
So here you can see that something which is actually zettl and predicted as one is zero, that is what
we were focusing on, that a non spam should not going to spam for.
So here we have actually misclassified fifty fifty six spam messages, that is something which is actually
a spam and do not spam, but that is not a problem because we are main target was this particular zettl,
which we have obtained.
So we are going to apply.
The regularisation barometer, see, do it so for this, we are again applying the creating the list
of CS, so here we are generating a list of CS from five hundred to two thousand.
And the school train is again blank school, this is again blank, so we are creating different Zeitels
values for this.
Now we are running SVM.
With different sea values, so here again, we are applying the SVC fit and then we are scoring these
different services and finding out the score and the precision scored from this.
And these are the values which we have generated from this.
Now, again, we are looking for the best decision to be one.
So we are looking at the model with the most, best precision, so that is data point nine nine five.
So this is the best model which does not produce any false positives.
So this was our goal.
So that is why we are taking this particular model.
Here are the monitors disposition is equal to one which we don't really have as of now.
So we can check that there are very few false positives so we can get the list of the best motives.
So this is the best way to deal with ninety nine point five percent or this decision and let this generate
the confusion, matrix or some confusion matrix can be obtained from matrix stored confusion matrix.
Using whitelist with respect to the speed or predict.
And we can get the predicted values, so here we have actually zero predicted as one as on day one and
those bombs classified as Norns from thirty seven.
So we have misclassified thirty seven spams and misclassified only one non spam.
So as a conclusion.
We can say that the best model is the one which is having ninety eight point three percent accuracy.
And it classifies every non spam message correctly.
So this is what we have obtained from this, so you can try different regularisation parameters and
try your own versions of these codes, try using the grid search KVI random.
So KVI for simplifying these implementations so that you do not waste time on running several loops
of codes which we have done here.
So you can try all that using grid.
So could.