So in the previous lecture, we discussed CVD in a generic sense, but we don't yet know how it applies
to an LP in this lecture.
We'll be discussing latent semantic analysis, which is precisely what we get when we apply CVD to a
term document matrix or a document term matrix.
But before we get to an LP specifically, let's discuss one more thing as we can do, which is reduce
redundant dimensions.
As you recall, I previously gave an example where we were looking at the height and weight of people.
We noted that these two dimensions are redundant because they can both be described by a single variable,
which is size.
To connect this back to the previous lecture, note how this is achieved by rotation.
We simply rotate the data or equivalently rotate the axes such that the white part goes along one axis
and the skinny part goes along the other axis.
Clearly, the white part is the part that represents our size variable.
And so this is the dimension we would decide to keep.
The other dimension looks like pure noise so it can be removed.
So that's how rotating allows us to remove redundancy.
Height and weight were redundant, and we combine these into a single variable called size using rotation
or in other words, as we'd.
Now, at the beginning of this section, I mentioned the concepts of synonyms and polygamy.
Personally, I don't think these are great ways to introduce SVOD or LSA.
And the reason is that most resources tend to introduce these concepts, but never explain exactly how
SVOD fixes these issues.
They usually just end up saying something like.
And therefore, SVOD appears to help.
This is, in my opinion, not satisfactory.
And this lecture will propose a simpler view based on what SVOD actually does.
As you've just seen, as we can reduce redundant dimensions, let's consider how this might arise in
NLP.
Suppose that we have two words which always co-occur, for instance, cat and feline, so every time
cat has a high value in the delivery of Matrix, feline does as well.
Another example is mitochondria ribosome in cytoplasm.
These are all parts of a biological cell.
But note that they are not synonymous.
They simply appear when the other words appear.
However, you can imagine how these two are redundant.
Any document about cells is likely to include words like mitochondria and ribosome and cytoplasm.
Every time the count for one of these is high, the count for the others are high as well.
And thus we have the same situation as height and weight.
These can all be reduced to single variables.
Mitochondria, ribosome and cytoplasm or words having to do with the cell cat and feline both just mean
cat.
And therefore, their dimensionality can be reduced.
And again, these are not necessarily synonyms.
They simply concur and note how this is the same situation as our height and weight example.
When the value of one is big, the value of the other is big as well.
In other words, it's the same kind of redundancy that as we can help get rid of.
So what do we get when we apply SVOD to a documentary matrix?
Well, it's very similar to topic modeling.
We didn't put a documentary matrix called X and we transform it into a document topic matrix called
Z.
As mentioned, our special SVB rotation will make it so that all the true data occurs in the left most
columns of Z.
Well, all the noise occurs to the right.
The code we use will actually cut off all the noise columns automatically.
So what we get back is just the relevant data.
Again, you can imagine these like hidden factors or topics.
So one topic might be regarding cats.
Another topic might be regarding cell biology and so forth.
One important concepts we haven't had time to discuss is that these hidden factors are actually ordered
in terms of importance.
So the first column is the most important topic.
The second column is the second most important topic and so forth.
So when you go to visualize your data by keeping only the first two columns, you are guaranteed that
you have the two most important columns.
This is unlike LDA and matrix factorization, where there is no such concept.
OK, so now that you know what SVT does and how it works in how to apply it to an LP, let's look at
how we'll do this in Python.
As usual, we can use Typekit learn, but note that SVT is actually a fundamental operation in linear
algebra, and it's not strictly machine learning.
And as such, you'll find that speed is actually a function in NumPy and CI Pi as well.
In fact, Saikia Learn uses this function behind the scenes when you learn about the math behind SVT.
You may find that this function is actually more useful in terms of using speed for other applications.
In any case, we begin as usual by creating an object of type truncated.
It says truncated because, as mentioned, we will be cutting off all the noise columns.
Note that this takes in an argument called end components just like LDA and an AMF.
In fact, just like with topic modeling, we as the user need to choose how many components we want
to keep.
Sometimes this will be very easy to choose.
For instance, if we want to visualize our data, then we should choose to.
Since our plot will be two dimensional.
But if we want to use speed as a pre processing step before spam detection, then we might want to choose
this value based on cross validation.
So how to choose the number of components will be context dependent.
There are more advanced concepts you might want to consider when choosing the number of components,
but those are outside the scope of this lecture.
The next step, as usual, is to call fit.
Now what is interesting about SVD is that it works pretty much exactly the same if your data is documents
by terms or terms by documents.
Normally in this course, we use documents by terms because we treat documents as samples in terms as
features.
But for the upcoming code example, we'll be using terms by documents.
This is because we'll be treating each term as a sample and we'll be reducing the dimensionality of
term vectors and then plotting those term vectors to see what patterns we can find.
As usual, we can then call the transform method to get back our transform data z, or we can do it
all in one step by calling Fit Transform.
Now again, as with LDA and OMF, the Z matrix has the shape end by K, where N is the number of samples,
and K is the number of hidden factors, which corresponds to end components above.
So in the case where we want to visualize our data, K would be two.
And by the way, note that this is another machine learning technique in which we have no targets and
we call transform instead of predict.
In other words, this is another instance of unsupervised learning.