So in this lecture, we are going to begin looking at our first and most simple method of converting
text into vectors.
This method is so simple that it doesn't even have a name, but if you want to give it a name, you
would simply call it counting.
As a side note, recognize that this technique is an instance of the bag of words approach, since we
will be ignoring the order of the words in each document.
OK, so suppose we're doing a simple classification task where we want to differentiate between the
documents about biology and documents about physics.
Now the first thing to keep in mind is that document is a very generic word.
This could mean anything depending on what your real world task is.
For example, you might want to categorize journal papers in which case one document would be one journal
paper.
Or you might want to categorize files on your desktop, in which case one document would be one file.
Of course, these files could still be journal papers, but they might also be for books, maybe even
simple text files, and so forth.
On the other hand, a document could simply refer to a single sentence.
For example, I might want to just look at a single sentence and classify whether it is positive or
negative.
An example of that might be news headlines.
So there is a wide range of possible document sizes from a single sentence all the way to a full book.
Now let's think about what our data would look like.
So we're going to start from a very general point of view where your data might be stored inside an
Excel spreadsheet, HSV or a pandas data frame, I'll assume that you all know your basic programming.
So if you had a bunch of data files lying around and you wanted to classify them, you would know how
to convert them into this format, since this isn't a machine learning task, but just a basic programming
task.
We would consider that a prerequisite to this course.
In any case, let's suppose that you were able to get your data into this format or that it was given
to you this way, which it could be.
In many cases, we can see that we have essentially two columns of data.
Column one is the text where each row is a document.
Column two is the label corresponding to the document beside it.
OK, so hopefully you find this format to be very simple, and you have a good idea about why your data
would be formatted in this way in the real world.
Note that we might also call column one the inputs to our model, while Column two consists of the targets.
Our goal in this lecture is to figure out how we will convert the documents in column one into a numerical
representation that would be fit for use in a machine learning model.
OK, so as mentioned, this method simply involves counting firstly, let's suppose that we know how
to determine the size of our vocabulary.
If you don't know how to do this yet, then please consider it as an exercise.
Basically, you would just need to live through every word in your text corpus.
Count up all the unique words, and that is your vocabulary size.
Suppose that we call our vocabulary size big V. It's the number of unique words in your training corpus.
OK, now the next thing to do is for each document in our text corpus.
We're going to create a vector of size v inside this vector.
We are going to count the number of times each word appears in the document.
Of course, this also means that we've assigned each position in the vector to correspond with a unique
word.
So, for example, the first position will contain the count for the word A.
The second position will contain the count for the word AA and so forth all the way up to the word zygote,
which we will assume is the last word in our corpus.
If we sort alphabetically.
Note that the words need not go in alphabetical order.
But for the purpose of this lecture, we'll just assume that they do for simplicity.
So let's go through a very simple example just to demonstrate how this works.
Let's suppose that we have a very simple vocabulary with just six words I like and hate eggs and cats.
Now, suppose that our documents are as follows Document one is I like eggs.
Document two is I hate cats.
Document three is I like cats and I like eggs.
Now let's convert these into vectors using counting now because their vocabulary size is six.
We're going to end up with three vectors each of size six one for each of our documents.
Let's suppose that the words will be ordered alphabetically, so the first position corresponds to and
the second to cats and so forth.
OK, so Document one would be converted to zero zero one zero one one.
This is because the eggs shows up once I shows up once and like, shows up once.
Document two would be converted to zero one zero one one zero.
This is because cats shows up once, he shows up once and he shows up once.
And finally, Document three would be converted to one one one zero two two.
This is because and it shows up once cats shows up, once eggs shows up.
Once I shows up twice and like, shows up twice as an exercise.
Please go through these yourself and confirm that they are correct or if I've made a mistake, which
is entirely possible.
Please let me know.
OK, so now that we know how to convert text into vectors by counting, let's return to our original
task, which was to classify between biology and physics documents.
Let's suppose that our counting scheme is very simple.
We only count the words mitochondria and gravity.
Now, if you don't know, the mitochondria is a component of the cell, and thus this would be associated
with biology.
On the other hand, gravity is usually involved with the study of physics.
So suppose that we did our accounting for all of our documents.
What might we expect to get?
Well, we would expect to get something like this.
Most of the biology documents appear along the mitochondria axis, while most of the physics documents
appear along the gravity axes.
Of course, sometimes biology documents might contain the word gravity and vice versa.
But clearly, there's a pattern here.
I can draw a neat line that separates the documents of the two classes.
That is to say, using these count vectors, I can now classify between documents about biology and
documents about physics.
So suppose I have a new document, and I'd like to automatically determine which class the document
belongs to without having to read the document in myself.
What I could do is use my computer program to count up the number of times each word appears, thus
converting it into a vector in this space.
Then I look at which side of the line does the vector fall on?
And that would be the class I predict for my new document.
OK, so hopefully that's pretty simple.
Now, there are a few things we have yet to discuss about converting text into vectors by counting.
These are all very practical issues.
That is, you wouldn't be able to code this up in Python without solving these issues first.
So the first issue is to recognize that text is represented in a computer by a string.
But our algorithm involves counting individual words, so the detail we have to consider is how do we
get a single string of text which contains a multiple words with punctuation and other complications
into single words that we can count?
This is a process called tokenization, which we will discuss shortly.
The second issue is this which we sort of alluded to earlier.
How do we know which component of our document vectors correspond to which word?
How do we keep track of this information?
Clearly, we're going to need some kind of mapping that will tell us which position corresponds with
which word.
Again, we will discuss this shortly, but I want you to start thinking now about how this might be
done.
The next issue I want to discuss in this lecture is how can we implement this in code?
Well, there are two ways to do this.
The first way which doesn't require much thinking simply uses the count vector riser class inside it.
Learn.
As usual, this proceeds in three steps.
The first step is to initialize account vector riser object.
The second step is to fit the count vector riser object to your training corpus.
You may also call Fit Transform, which will give you back the matrix of counts and fit the model at
the same time.
The third step is to transform the test corpus.
Remember that in general, we don't want to fit on the whole data set at the same time, since the object
may store parameters that depend on the data you passed in.
For example, if we have a word that appears in the test set but not the train set, you wouldn't want
to include that in your count vectors as your model wouldn't know what to do with those words anyway,
since we never saw them during the training process.
Now, the second way to implement are counting method is to simply implemented ourselves using Python
and potentially Nampai or CPI, although we won't look at how to do this in code.
You are strongly encouraged to do it on your own as an exercise.
It's easy enough that you can do it by yourself without assistance, and it's a good way to gain experience
with writing code.
So please give this a try.
As a side note, the reason you would want to use PSI Pi instead of Nampai is because it has smarts
matrices.
Suppose that we have any documents and a vocabulary size of V.
Our data will ultimately be stored in a matrix of size N by V and rows and V columns.
Now, it's pretty likely that most documents will not contain most words.
That's just the nature of language.
As an example, suppose that our data is a collection of articles from Wikipedia, and our vocabulary
is pretty much all of the English language.
Well, clearly any single Wikipedia article will not contain every word in the English language.
More likely, most, if not all, Wikipedia articles will only contain a very small fraction of the
possible words in the English language, and as such, our final end by V Matrix will consist mostly
of zeros.
Now, it turns out that there are very efficient ways to store matrices when they consist of mostly
zeros.
These are called sparse matrices.
At this point, the details aren't very important, but it's just something to be aware of.
I recommend for now to read the documentation if you want to learn more.
But at this time, it's not strictly necessary for what we're going to do.
So if you do the exercise of implementing the count vector as a yourself, you may want to look into
using these.
OK, so the final topic I want to touch on in this lecture is the concept of vector normalization.
Now suppose that some of the documents in our dataset are very long, while others are very short.
Our results in count vectors might have large size disparities simply because long documents contain
more words.
More words means higher counts.
Higher counts means larger vectors.
And this might be a bad thing since, as mentioned, we want similar documents to be close to each other
in vector space.
So there are two ways to normalize vectors.
Firstly, note that our count vectors always contain at non-negative numbers, either the value is zero
because the word didn't appear or the value is greater than zero if the word did appear.
But we can have a negative count for any word.
So one way to normalize a vector is to make it have an El two norm of one.
As you recall from your high school math studies, this involves dividing by the square root of the
sum of squares of each element.
This is basically the length of the vector.
The second method is to simply divide by the Sun so that the sum of each element in every document vector
is one.
As you'll see, when we study probabilistic models, this gives us something like the probability that
each word appears in the document.
So that's one way to interpret this method of normalization.
Note that an equivalent method is to make the L1 norm of the count vector one, the reason why this
is equivalent is because we already know that all the counts are non-negative.
Technically, the L1 norm is the sum of all the absolute values of each element.
But since we already know all the values are not negative, the absolute values are equal to the original
values.
And this is just equal to dividing by the sum.
So either way, this will ensure that the sum of all the elements is one.
As a final note, I want to mention that the count victories are inside, you learn, does not implement
any normalization.
However, TF IDF does, which we will learn about later.
So if you want to build count vectors with normalization, you would use to fight off and not the count
vector riser.