Didn't know we have discussed about structured data, this type of data had different columns, these
columns contain categorical numerical and ordinal values, and along with that, we used to have either
a target value, which was a class form, or it was a numerical form which would help finding out a
regression.
Now, let's see, instead of having a regression problem and instead of having a structured data, let's
say we're working on a fiction legacy.
The did not have such data.
We had data which had a lot of text present in it.
Now, we cannot really convoyed this entire text in two different columns.
And even if we were able to convert this into different columns, the machine would not be able to find
out the meaning of different words in the text.
So how do we actually handle this particular data and how we can transform this particular data such
that we can apply models on top of that?
This is known as natural language processing.
How we will process this entire text is what we would learn next.
So let's go ahead.
Now, what are the benefits of natural language processing?
The benefit of natural language processing are in natural language processing can be leveraged by companies
to improve the efficiency of documentation process, to improve the accuracy of documentation, to identify
the most pertinent information from large devices legacy, the an organization who wanted to find out
which customers were satisfied and what number of customers were not really satisfied with the services.
Now for that, we wanted to have a look at.
So what we do is we take a complete collection of the tweets from our users and try to find out which
user has tweeted something positive about the organization and which one has tweeted something negative
about the organization.
Then after that, from the negative feedback, what we try to do is we can seek out those specific keywords
which were related to some products or services which our organization was providing.
Let us say we make shoes and shirts so we can find out the negative comments from the tweets.
Now, let's say we found out that all the shoes have some negative tweets.
So from that particular text, we can.
Extract those specific words, which would Volchek on what is actually wrong with our shoes.
So this is something what we can achieve from natural language processing, using the machines, which
does not really understand what languages.
So let us work further on it now, what is the next mining xed mining refers to the process of deriving
high quality information from the text.
The overarching goal is essentially to build the biggest into data for analysis via application of natural
language processing.
So instead of having that structural data, we now have the actual data.
Now our task is to convert the actual data into that structured data, which we have been walking along
with.
Let us talk about what structured data is and what unstructured data is, the structured data is any
data which resides in a fixed field within a record or a fight, for example, data in a database table
which is easy to enter, easy to store and analyze.
We have been working with such kind of data for some time now.
The unstructured data does not really reside in a traditional database.
For example, it means video or audio files, webpages, presentations.
These are all difficult and costly to analyze.
So to our rescue, we have natural language processing now there are a few basic concepts which I want
you to understand.
First one is tokenization.
Tokenization is the process of converting text into tokens.
So let us say I have a sentence.
My house is very beautiful.
So out of this sentence, I have to create several tokens, so the tokens would be my the next token
would be house then is then very then beautiful.
So each and every word in this text is actually a token.
So tokens are words or entities present in the text and tokenization is the process of converting the
entire text into small tokens.
Different next objects could be a sentence, a phrase, a word or an entire article.
Now, what we actually do in MLP is the segment, the words and sentences into different buckets.
And after that, we pre-process this text.
The preprocessing of this text include removal of Stopford, then stemming or limitation, and finally
then quoting these text words in two different vectors.
Now, let us see what this is.
So like I say, we have this particular text.
The text is my Red Sox are the prettiest socks in the country.
No other Red Sox are pretty in the nation.
Now, here, if you consider we have several, quote, commas and full stops, presenta, you know,
if we consider these sentences, these sentences also have words.
But she does not really are of importance.
For example, in the are in, though, these are different words which we don't really need, because
even if we say my Red Sox prettiest Sox country, no other Red Sox nation, now, this would kind of
convey the same thing, but only we would be rejecting certain words which are of not importance.
So the first step is to convert these and remove those unwanted things, so we remove this dump only
punctuations.
So now we have my Red Sox for Sox country, nor the Red Sox nation.
Now, the first process which we apply is now.
What is STEM stemming is the process of reducing the world's generally modified or derived through their
word STEM or group form.
The objective of STEM is to reduce the related words for the same stem, even if the STEM is not a dictionary
word.
Now we are converting different words into some form, which is not actually a dictionary word, but
something which is a broad word.
We are removing the suffix from this word.
OK, so let us see an example.
So here we have words from English language, beautiful and beautifully.
Now if you see the meaning from beautiful and beautifully is a little different, but still because
the root would that is be a unity is common in both of these words.
These are stemmed to be while the words good, better and best are having the same meaning, but because
they are not having a suffix kind of a structure where same word is extended by a suffix, these are
converted into good, better, best only they are not converted by stimming.
So stemming has no impact on these kind of words who do not share a same word stem.
So in this particular example, my red blue socks, prettiest and prettier, both of these have the
same wood, so they will be converted into that would be the FBI.
Now, what limited organization does is the process of reducing a group of words into the Alema or dictionary
form, it takes into account the things like parts of speech and it can words, and it also considers
the context behind a particular word.
So, for example, we have a sentence with beautiful and beautifully.
So they will be limited to beautiful and beautifully respectively, because the meaning behind them
is different because the part of speech is different.
Why is this a good, better and best?
They will be limited to good, good and good because the context behind good, better, best is the
same.
All three words are actually meaning good and that with different intensities.
So it simply kind words, all three words into just one single word.
So here, when we apply limited information to this particular sentence, the prettiest I reviewed because
both of these words are similar in context and are just differing in the intensity, they are converted
into proving via the word country and nation because they have similar meaning, they will be converted
into country.
So this is what limited accusation and stemming is now stammers typically easier to implement.
And run faster and the reduced accuracy may not matter for some application.
So if you want to apply limitation or stimming, it is always preferable to apply limitation if you
are looking for a better result.
But in case we do not look for better results, but for a faster implementation or a simpler implementation,
then we can go for stimming.
Now, after they have applied, stemming all improvisation, what we do is we can watch these words
into a video form.
So there are two types of ways of doing this.
One is converting it into account features.
So what are called features?
So let's say we have those words so we can create these kind of features, these kind of columns where
each word we have the frequency of its utterance.
So he in the sentence, which we were discussing my game once that was occurring twice, blew up twice,
sucks up twice already occurred twice country twice.
No other one and other cold ones.
Here in these sentences, the Red Dog, the ones red ones, the old ones and all of that are so you
can understand like here drink and eat.
So here red has frequency one, cat has frequency one, it has frequency one, and all other values
are zero.
So this kind of a structure would be created.
Now, let us try to understand, this is a scenario when we have just a few words like the Red Dog guac
and eat only such a small number of words.
Only six words created a huge matrix like this.
Now, if we are working with huge stakes, then the matrix which would be created would be very large.
So that is the reason why the force step we did was to remove the Stopford and the punctuations so that
we could reduce the size of the Matrix, which would be great.
Now, another thing is that when we are creating this matrix, we do not want words to occur very frequently
or the words which occur very less number of times.
So for that, instead of creating a simple counterweight to what we can do, is we can create the idea
of vector.
Which is the frequency, inverse document frequency.
Now, what this is, is it will try to keep only those words which are not very frequently occurring
and which are not very specific.
So it will basically keep in consideration the tone frequency, the frequency is the number of occurrences
of the storm.
Into one particular document, I'm the product.
Of it with the inverse document frequency is total number of documents divided by the number of documents
containing that specific book.
So this is the formula which is associated with the idea which is applied so that all future space can
be created.
And once we have either our own features or the of features, we can attach the target column with it
and use it like a simple classification model or just the way how we want to use.
Similarly, like we have been doing it with the structured data, the same things can be implemented,
implemented with this unstructured data, which now has been transformed into a structured form.
Let us have a look at the gold walkthrough in the next session.