So in this lecture, we will be looking at how to implement sentiment analysis.
We'll begin by downloading our data set, as we normally do.
Note that this data set is a data set of airline tweets.
So these are tweets that Twitter users are making to an airline.
Based on this, you might be able to guess whether most of these tweets are positive, negative or neutral.
The next step is to import everything we need for this notebook.
Most of this stuff you should have seen before, so there's not much to say.
The next step is to load in our data using PD that reads ESV Note that I've given this the variable
name of ADF underscore since we'll be filtering out this data frame shortly.
The next step is to call the head command to see what our data frame looks like.
OK, so notice that we have many columns, most of which are not needed.
The columns that we will make use of are the sentiment column and the column called text, which contains
the tweet itself.
Please feel free to look through this yourself to see if there's anything else you'd like to use.
One interesting column that would have been interesting to use is the sentiment confidence.
This one is interesting since some models allow you to apply weights to each data sample.
By default, these weights are usually uniform.
But in this case, since the confidence score differs from sample to sample, it might make sense to
weight the samples by this confidence.
In that way, the model will pay less attention to samples with low confidence.
In any case, we are not doing that in this notebook, but it might be a nice exercise for you to try
by yourself.
The next step is to filter out the columns we want, which are airline sentiment and text, as mentioned.
Note that we will call the result death.
The next step is to call the head function to see what our new data frame looks like.
So as you can see, we now have the usual two column data dataframe that we expect for this type of
problem.
The next step is to plot a histogram of the labels to check whether or not we have any class imbalance.
OK, so it does appear that we have class imbalance, in particular, the negative class is overrepresented.
This makes sense since users often use Twitter to complain to companies about their bad experiences.
So because of this class imbalance, we should apply techniques like planning the confusion matrix as
well as computing there you see.
The next step is to convert the class labels into the conventional format, which are integers from
zero up to minus one.
As you can see, this simply means I'm going to create a dictionary mapping from string label to integer.
Once I've done that, I can call the map function on the data frame column, which I want to apply this
mapping to.
The next step is to call the after head again to check that the targets were created correctly.
OK, so this confirms that our targets look as expected.
The next step is to split our data into train and test.
The next step is to call the head function on our train data set to make sure that it looks like we
expect.
As you can see, the rows have been shuffled, which is what we expect.
The next step is to create a TF IDF vector as your object.
Note that I've said max features to two thousand, which will limit the vocabulary size of our model.
The next step is to fit our veterans are to the training data and to transform it into X train.
The next step is to print X train again to confirm that we get back a matrix of TF IDF values.
So as expected, we get a sparse matrix with two thousand columns.
The next step is to transform the test data into a test.
The next step is to assign why weight training, why test, which come from the target column of their
respective data frames.
The next step is to build a logistic regression instance, train it and check our models performance
note that have set max error to five 500 since.
As I recall, this led to a warning that the training process had not converged with the default values.
As you know, the next step is to call fit and then to go score on both the training test sets.
OK, so as you can see, we get about 85 percent on the train set in about 80 percent on the test set.
But remember, these numbers should not be looked at in isolation.
Since we have imbalanced classes.
The next step is to check the AUC.
Now, as you recall, the AUC is defined as the area under the ROTC, which is defined in terms of binary
detection.
However, recall that in this problem, we have three classes, so let's proceed as we normally would
by calling model predicts proper.
Now, normally for two classes, the return value would be an end by two array and we would choose the
column at index one.
So you can see how I've commented that out.
In this case, we're actually going to keep all the columns.
This is necessary for the next step, which is to call the function RC AUC score.
You'll notice that I've passed in an additional argument called the multiclass and I've said it to over
you.
So Oyo stands for one versus one.
This is opposed to one versus rest.
Essentially, these are two different strategies to build a multi class model from a binary model.
Now, it's not my goal in this lecture to go off on a tangent about how that's done at this point.
But if you want a deeper understanding, please feel free to look it up.
It is, in my opinion, kind of interesting.
The main idea you should keep in mind is that there are ways to convert binary metrics into multiclass
metrics.
OK, so interestingly, the AUC values are much higher than the accuracy's.
OK, so the next step is to work on plotting the confusion matrix.
So we'll start by getting our model predictions this time as class labels instead of probabilities.
The next step is to call the confusion matrix function.
Note that have said normalized the true, which will make it so that each row of the matrix sums to
one.
This is helpful to interpret the output, since the number of samples in each class vary by a lot.
OK, so as you can see, we get back a three by three matrix.
As expected.
The next step is to plot the confusion matrix, as we had previously done.
Recall that the reason why we are not using psyche to learn even though it has this functionality,
is because they are in the middle of transitioning to a new version and this API is going to change.
Luckily, this method is just as good.
OK, so as you can see, our model does very well on the negative class, getting about 96 percent of
samples correct for the positive class.
We only get about 70 percent correct.
And interestingly, when we do not predict the positive class correctly, we actually end up predicting
negative rather than neutral.
So it doesn't seem to be a matter of simply confusing positive tweets with neutral tweets.
Now, if we look at the neutral class, the performance here is the worst at about sixty five percent
in this case.
Again, when we are incorrect, we most often predict a negative, which happens about 30 percent of
the time.
Thus, our model appears to have a bias towards the negative class.
This makes sense since it is the overrepresented class.
So the next step is to plot the confusion matrix for the test set against setting normalized the true.
OK, so in this case, the results reflect what we see for the train set, except that the performance
is a bit worse.
In particular, we now only get about 64 percent of the positive tweets correct and about 53 percent
of the neutral tweets.
And again, there is a large bias towards the negative class.
Now, since this lecture has been quite long so far, we're going to take a break in the next lecture.
We'll continue with looking at the binary case, along with interpreting the model weights.