And in this session, we will discuss about my base implementation.
So for this particular implementation, I have chosen the Women's Clothing E-commerce Review's dataset.
So for this particular dataset, again, I would be exploring the age group of females, the age group
which bought what kind of growth?
The age group which bought what kind of product based on class name.
The department has what number of percentage of class names then which division?
The different division, different department names, then the column frequency of the woods.
And all of these details would be discussed inside this.
So I hope you would understand all the codes, as we have already discussed about an LP.
So this would give you a more insight on how we implement data or data transformations and analysis
on the actual data.
So let us go ahead and the first thing which we will be working on with would be the importing of the
libraries.
So the libraries, which we would be working with are Fondas, number seven, McLauchlan and find us
providing.
Now, the data which we will be importing is, again, the women's clothing data, and it does not have
the index volume.
So now we have imported the data.
We don't read CSFI and this is the data frame which we have created out of it.
Now we have created a list of columns.
So this is the column contain list, which has different column list.
That is clothing like the age title review rating, recommended positive feedback division name, department
name and last name.
So these are the different column names which we are going to assign to this particular dataset.
So the how we provide the column names is by using the DOT data frame.
I'm in a data frame.
We will give the data under the data.
Then we would give the column names under the columns, then we can use different informal visualize
different details about these columns.
So we have total twenty three thousand four hundred twenty six of one hundred eighty six rows of data.
And.
The first one being the clothing line is integer 60 40.
Then we have Ege, which is again in DNA, then we have Bitel, which is object.
We have reviewed text which has objects.
Then we have reefing, which is in 60 for recommended index, the in 60 for positive feedback in 60
for division name, object, department name, object and class name object.
The entire memory used for this data is more than one point lead and now the say it has twenty three
thousand four hundred and eighty six rows of data and there are nine columns in total.
Now some of these entries are missing like title division named Batmen, name and last name.
So let's, first of all, view the data using the dopehead.
So this is the data which we have, which is having the clothing need, which is the cloth, Heidi,
then the age of the age group of the person, then the title, which is what?
The title of this particular review, which we have.
Then the text of the review, then the writing that the person has given, and if someone has recommended
this particular plot or not, if they have recommended it says one, if someone has not recommended,
then it says zero.
Last is the positive feedback on how many positive feedback has this review received.
Then we have the division name.
That is what kind of plot is it?
Is it in intimate or gender or gender?
What kind of typewriters then?
The department name that is in it is intimate.
It is dresses.
It is bottoms.
It stops.
Then that is another class name which is again dresses, finds or blouses.
What is it.
Now, let us have a look at the new medical columns so the new medical columns of clothing, idy age
rating, recommended index and positive feedback so we can see that we have seventy five percent.
I'll add one zero eight maximum one to zero five, because it doesn't really matter.
Then we have age.
Each has data from it being then the twenty five percentile at 30, 40, 50 percent, Vaillant's forty
one seventy five percent, Bilad 52 and maximum at ninety nine.
So there seems to be some kind of outliers present in the age of the people who are reviewing.
Then we have the rating which ranges from one to five.
And you can see that the twenty five percentile rating is actually four.
So this means that most of the people actually give a high rating.
Most of the people are giving the rating, which is four.
And there are very few number of people who are giving less than four rating.
That is only twenty five percent of people are giving less than four rating if this twenty five percent
percentile has the minimum value being for now.
Next is the recommendation index, so you can see the minimum value is zero and twenty five percentile
is one, which again means that most of the clothes are actually recommended here.
Then we have positive feedback so we can see that they're at 50 percent showing the positive feedback
on this one.
And then it is above.
So below 50 percent.
Dalitz is usually zero.
Now, let us check if there is any type of correlation between the user's rating and the review length
of the reviews.
So what we can do is we have taken the data frame review fixed and we have created a new column which
is review fixed.
And in it we have added the type string.
So we have converted the review text to string.
And in the review length, we have just applied the lente on this review based so that we can get the
length of the review.
Then we are just creating the grid and we are playing the game up and plotting the histogram of the
review linked with the Vinces size 50.
So let us have a look at this.
So here we have the ratings for rating one.
Four, we have the review length here, so you can see that reading one of here, the review length
does not really matter much.
It is almost the same.
And if you go further, that is one big if you see here and in all the plots, there are these big peaks
which are available now.
These peaks are usually available.
You can see of these four or five, the peaks are higher.
So it means that there are longer reviews.
If the readings are higher.
Now, from above chart, we can see that user gave five rating of only.
There were more number of five readings.
And in fact, there are no there are less number of users who gave rating one and two.
You can see there is very little amount of data.
And you.
Now, let us plot another road to you now, this plot we are creating between the rating and the review,
Len.
So if you compare this, is that a violent.
And these are the ratings, so you can see that the.
There is not much difference, the review length is higher for ratings three and four.
So from above, we can go through that routine for entry, have more length interviews.
These are higher in value, so these have more number of reviews, more lengthy reviews provided.
Next, let us have a look at the ratings so we will go by ratings with the means so we can see here.
Four of you will be finding out the correlation between the different ratings, so we have clothing
in which clothing indeed there is a more specific correlation with age.
There is a negative high correlation with the clothing, Islay.
Then recommended index, there is a low correlation, the violent has a high correlation here we have
age and the violent, which has a negatively high negative, strong correlation.
So here we have clothing, Ivan Ege, then after that, we have positive feedback on the recommendation
index, which has a negative high correlation.
Then after that, we have.
These are the only important ones in this.
Now, let us have a look at this, this is the coalition, Jack.
Now, here we can see that there is not much high correlation among the columns now, the columns like
review length and the columns like age and positive feedback have a strong correlation of and the numbers
are zero point three or nine three in negative indicates that it is nowhere correlated with the Ruland.
So as the age grows, the length of the view decreases.
So this is the data set, which we have again.
Now, we will group the data with respect to the rating and the data frame of GOLOMB each, and we are
just creating our values from zero to hundred.
With a gap of 10 and we are just on stacking and plotting the budget for the season.
So here you can see that for these different age groups, we have compared the ratings.
So you can see that the blue one being the rating.
So you can see that the five rating is given by the most of the people age between 30 to 40 are giving
very high ratings.
And usually most of the people are giving high ratings rating five and rating for there are very few
number of people who are actually giving lower ratings.
So from above our flawed, we can see that the age group, 10 to 20 gave less ratings, less number
of ratings were provided by them.
And it is obvious that in this age group, teenagers generally don't care about online shopping interviews.
The age group 30 to 40 give five rating as compared to all other age groups.
And in fact, this is the age group who gave most reviews and ratings.
And similarly, the age group of 70 did not care about the online shopping stuff.
So this is what this blog displays.
And as told earlier, the number of reviews, the positive reviews are higher in all the age groups.
Now we will plot another chart between the department name and the age with the another budget.
So here we have created the vacancy that the reviews because the number of reviews are less for these
age groups.
So we don't have much information about them.
But for the age groups from 20 to 70, we do have the ability.
So you can see that the number of Thorp's is actually having a high number of Thorp's or something,
which has been sold mostly and reviewed about mostly.
And then the second number, we have dresses.
This, again, has more number of things, and after that we have for the blues that does the bottoms,
so the most number of reviews were given about the tops, then followed by the dresses and then followed
by the.
Waddle's.
So in the above, I applaud, I want to concentrate on the department and the age group, the female
from 20 to 70 age were more active and more of the stuff on line from the above.
But we conclude that females were more focused on Phelps and his department and somewhat focused on
boards, too, but not that much.
They were less concentrated on training department.
So because that's not really visible here.
So.
Next, we would try to find a relationship between class name and the age, so we will have another
look for that.
So here you can see that the nets have more number of seals.
And similarly, the red one, which is the addresses.
So these are the most famous last names.
Then we have creating or we are creating on a block between the department name.
And the guns, so here you can see that the most number of selling department is bulb's, followed by
addresses followed by Boredoms, then followed by demesne, then followed by a and train has the minimum
amount.
So above shows that the maximum entries for top, which is around ten thousand five hundred, and then
this department is having around six thousand entries.
So above board shows that the maximum number of entries from Thorp's.
Now, we will compare the division names with respect to the guns, so the code which I have written
for this again, is again the simple ones, which we have already discussed.
We simply use the C one library for this.
So it is violate Bolex neighborliness and escorted by law.
So I don't think there would be much problem understanding this.
So let's go ahead and you can see the columns for general is very high, that is around fourteen thousand.
Then the columns for Generativity Division is around eight thousand and the lowest one is for intimate.
That is around two thousand.
So in our data, there are three divisions that as General General David and the general division products
were most sold out as compared to the general, the intimate.
Now we will start walking with the actual data, so from collections I have in Dade County and from
the analytical library, I am importing the corpus.
Now this corpus has these stop words in it.
So again, repeating what the stop would stop words are the words which don't really provide much meaning
to our data, textual data.
So we will try to remove them because once we remove the stop words, then a huge chunk of words get
removed from the set of tokens, which we will be creating.
So next is the organizer, which we will be importing from and get organized.
Now, this is a regular expression organizer.
And next, we will import the sentiment organizer and devoted organizer.
So these are different types of organizers which we will be using.
And finally, we are importing the award cloud.
And word cloud stop words from the word cloud library.
You can see what word cloud that's for us.
So now what we are doing is we are creating a variable that is stop and.
And we are converting the list of list into next.
So we are creating a particular video which will hold the column Adreview Fixed.
And.
It will.
Contain the values separated by the space, then after that, we will be removing the punctuations numbers
and returning the list of only words, basically we are selecting anything.
Which is having the words in it, and after that, they are removing all the stop words by simply getting
the list of words, list of stop words from English language.
So stop start words and get all the words from the English language here.
And we have got the English Stopford now, what we will do is we will tokenized the words which we will
get, so we apply the word to organize and put the words into B of these into words organize so we will
get the word Duplantier.
And after that, we are just applying the law, which will take the word from the word tokens.
And check if the word belongs to stop words, if not, then it will go on and put it into those sentences.
After that, we are simply checking the words and upending different words from also from the word open
to the sentence.
And here we are removing the characters which have lent less than to.
So whatever words we have, which have the Salento of the word listen to, we are removing them and
keeping only the words which have lent more than to.
So here we have just filtered with the land of word rather than two.
And I did all those too without a single.
Then we are removing the numbers from this particular data, so we are simply checking for walked into
the room without a single directo and checking if the word is not numeric, then we are putting the
word in to the clean data.
Now, after this, we are simply calculating the frequency distributions or how we are doing this,
we are using the sequence of distribution of function, which will get Deklin data inside this on the
word distribution.
Finally, here we are creating a data frame with the word distribution and getting the top hundred of
words from the most common words from it, here we have defined development to be hundreds.
So here we are getting the top hundred words from this and creating the columns to be board and frequency.
Now we are just plotting this, using the block, so here we have the words so that all words are addresses
love size to fit like an.
So this above my blog titled Subsequence View of the World in the review column, and the word race
appears more in the text next to this, the word love comes second.
And it is an indicator of a positive that is more number of reviews are positive, which we already
have, that the most of the reviews which have been given how high the rating for more than four from
the twenty fifth person died.
So from that only we can see that that is a lot of positive comments coming in.
So here's what we are doing is we are generating the word cloud, so forward cloud, we are just defining
the details of the word clouthier.
So we are saying we want to see good and for the good.
We are saying we want the word cloud in the word.
Now we want to give the background color.
And the maximum number of words as thousand and maximum form signs as 50, and then we are just reading
the words the like, we are just combining the data with US space and then we are generating the word
cloud.
So this is the word cloud which we have generated with the black cloud and most use it.
So these are the word clouds.
So here you can see what is a word cloud, what cloud is basically a combination of all the words which
are present in the data.
So the most frequently occurring words will be placed in this chart and this horizontal and vertical
words would be written here.
The size of the word actually shows how frequently the word has been used.
So you can see that the dress, the word dress is very large in size.
So this is the most frequently used word.
Then we have the word look, we have the word love, which is larger in size than we have the fabric,
pink color.
Great.
So sweaters.
So these are the few words which are very frequently used in.
Most used, while we have another word like you, nice, dried fit, thighed, blouse, walk the design
the day, love breezeways, true size but front.
So these are different other words which have been used in the reviews but are lesser frequently happening.
That is why the sizes are so small.
So.
This is what is the biggest volume now.
Next, what we will be doing is we will be analyzing the sentiment of the Dow, which we have so far,
analyzing the sentiments we are importing.
Fluff blog from the X Blob Library.
So we are creating a blank list that is a blob list description in this of another name, which is date
of name review string, which contains the date of names review.
So all the review fakes would be saved as string.
Then we are going through the entire review list, one by one, rule by rule and the.
Creating the explosion of this and this is the block value now from the globalist, we are upending
this list, which we have created in this.
We are upending the sentiment polarity to it.
So the sentiment polarity and sentiment subjectivity will actually have the details of the sentiments
of a particular day.
That is how much positive the comment is, how much negative the comment is, how much new value of
the new is present in this particular review.
So those things will be given by the sentiment, polarity and subjectivity.
So we have this data frame.
So in the data frame, we have given the bloodless description and given the the view, the sentiment
and the polarity value.
So we have created this one function, which is checking if the polarity description is sentiment,
it is if it is, if this value is greater than zero, then we give it a positive review.
If the polarity is equal to zero, then we give it as a neutral review.
If the value is neither greater than zero, not equal to then it means that it is less than zero.
So that means it is a negative review.
And based on that, we are simply.
Created a lot of effort that does account a lot of this, so this is the plot which shows that that
is more than twenty thousand positive reviews and less than twenty five hundred negative and neutral
reviews present.
Now, what you can do is you can explore the explore library and see how polarity works, how subjectivity
works.
So these are different things which are present in sentiment analysis.
So you can use these.
I'm trying to understand about these how these can be used.
This is one example of using sentiment analysis.
You can use several other movies and make your own implementations.
Now, here's what we are doing is we are just getting the details and getting the plotting the for the
positive reviews.
So for positive reviews, the word cloud has words like love, dress of fabric, ball, fake look really
true.
Sighs Well, so these are some positive words from the positive reviews.
Next, we will test for the negative reviews, so here we have the negative reviews, so it has a little.
Quality disappointed.
Look small, so all the material.
Then we have lent you one at least, so these are different words which we can find from the negative
reviews so you can similarly plot a word cloud for the neutral reviews.
So there are different ways how you can use the sentiment analysis.
Next is.
We're just getting the details we are creating under the function that this process, so here we are
simply processing the data via removing the punctuations.
So we are simply getting the words from the board in reviews and checking if the word is not a string
punctuation.
So this is a string dot punctuation is used to check if the word is a punctuation or not.
So.
It will just check and put all the non punctuations in, you know, bunk and join it with the space,
then we are just returning this splitted one with the non punctuations and by removing the Stopford
from it.
So we are just removing the punctuations and stop boards from this day.
DITZEN Now we have the data here.
This is these are the words which we have received.
So the words are absolute wonderful.
Silkies ACSI Comfortable love is so pretty.
So these are different words which we have in the review text now from which the punctuations have been
removed from the English word has been removed.
So this is one type of implementation we have on another type of implementation during the end of the
session when we discussed how we can process the data.
So I'm just showing different types of implementations.
So get in so that you can try different implementations and decide which one you like.
What would you like to do?
And based on the problem you have, you can decide what implementations you want to do.
So we are just for once we have this data.
We will vectorized the data.
So here we are using the victories in the implementation of we had used the idea of victimiser.
So for this one, I've chosen the conflict arising.
So let us use the common victimiser now.
So we are just simply separating the data and creating the review data frame and the video frame for
this.
So the extra data frame contains the review text and the Vivan by our data frame contains the three
things.
So that is what we want to do from the interview, because we want to classify the ratings here, right.
So from the Eskil on feature extraction, we get the victimiser.
So the first thing, what we will be doing is to fit the gown's victimiser on the.
Training it right, the first thing which we do is the first debate that I said, no matter which vectorized
that it is, if it is a victimiser or it is a video victimiser, we will fight the way that I use it
on the input data, on the training data.
And then we will use the.
Transform function to actually transforming the beat.
This is what we do because of any future, also, we will have to transform the data.
This we have already discussed that because we have different though means we have different type of
words based on the requirement.
So if we are talking about e-mail spam, then we will have different type of words which will need to
be transformed and there will be a different type of word list in know in a chemistry project or there
would be a different type of word in the banking domain.
So based on the words which are actually used, we will be, first of all, fitting the transformer
that is draining the transformer and then actually transforming.
So that's what we do here.
We create this VW transformer and then use this V or W transformer on the vocabulary.
So how would we do that?
We will simply do the UAW plan to transform, to transform the X reviewing X reviews.
Now, we will be splitting the data now, how we split the data, we can use any of the methods, the
one being three in the split, we can use this also.
And another method is the.
Cross validation, that is, of greater TV or random thoughts TV, so we could have used any method.
I am using one method.
Your task is while practicing dry, brittle TV or cross-validation, so you have to try all different
variants so that you learn all the ways of doing it.
So we have included the Skillern model selection and the three best split.
Now, what will we be doing is we will split the data extra and why I'm here, that this size is zero
point three, which means that there will be 10 percent data in my testing dataset and 70 percent data
in my training dataset.
And from this eye opening, the extreme best light rain and witnessed.
Now we've got.
Predict the reading of the review, we will use a knife based machine learning algorithm.
So for that, we will be importing the multinomial name based from the name based.
Library from the library.
So we are creating the object of Navys and putting the training data into it once we have secured the
data into it.
We will use and we do not predict to predict the values of the FESTE data.
Now we get the predict values.
We will compare these predict values with the actual very best values.
So what we are doing, we are importing the Confusion, Matrix and Classification Report and we are
comparing the lightest and predict using the classic confusion, matrix and classification reports so
that these could be generated.
So these are the.
Confusion Friedrichsen classification report, which we have got.
So here you can see that.
The precision for the first class is zero point eight six one, the fifth one is zero point nine six
and so on.
So our model has achieved on ninety five percent efficiency here.
It means that the business can predict the users like the product or not.
Now we will test the model with the data.
So this is the review text the third one.
So we are checking if the rating is positive or not.
So let us check.
So it says, I love, love, love this jumpsuit.
It is fun for the families.
Every time a word, I get nothing but grief.
So first I want to test with the positive review.
So I have chosen the above rating.
So this one has a rating five.
So let us see.
So we will transform this using the transformer.
Then after transforming, we get the rating positive transform and we will now predicted using the Navy's
not predict what is the value of the predicted one.
It is five.
So it is giving the correct predicted value.
Now we will test a negative comment.
So this is a negative comment which we have got and we will predict this.
This says three times soon to spawn one huge.
It is very cheap.
So I cut them out.
Then the train left behind was Plastiki and even more.
Ichi, how can you make an intimate with such Dag's not comfortable and different views.
So we get the rating negative control and we just transform the data and then predict the value.
And it comes out to be what?
So now we will predict if the item will be recommended or not.
So what we are doing, we are getting the expert recommend and I recommend and again, we will do the
same thing.
We will make divisor on top of it because, again, of the very predictive value and the of the predictive
value, these will have no different type of words.
But as in trade.
So we will again analyze, we will again of foot another vectorized on it and then transform using this
vectorized.
I do the same thing that this split the data into a strange split with the naval base and we will follow
the Navy and then find out the probability, the predict, and we don't predict the value predictor.
And then we will get the confusion, matrix and classification report out of this.
Now, for this one, the model predicted eighty seven percent deficiency.
So now we test again for the leading positive and negative.
So for a positive rating, it gives one that is it is recommended.
So the user has actually recommended this now in the above block, the model predicted correctly, that
is, it was a positive review, so it was recommended now for the negative review.
It is giving zero.
That is, it will not be recommended.
So like this, you can use any classification algorithm to implement actual data classification.
And Nijhuis is a very good algorithm to implement this.
You can implement this using SVM a little while.
You will be working with SVM.
We will again implement a different kind of data.
So that you can get hold of how we shall transform the data and learn more from it.