So in this lecture, we will be introducing the intuition behind singular value decomposition, also
known as SVT.
There are three basic concepts you want to keep in mind for this lecture.
Number one seed is useful for visualizing your data.
Number two as SVT is useful for reducing a dimensionality.
And Number three as FEED works by finding the best rotation of your data points.
This lecture will look at the relationships between these three concepts, and we'll look at a hopefully
intuitive picture that explains how this works.
OK, so let's first talk about why doing these things are even useful.
Let's begin with reducing dimensionality.
We know that in NLP, our data is pretty much always high dimensional.
This is because we work with document term matrices like TFI Taf, where the number of terms can be
in the tens of thousands or even millions for many datasets will have a number of features, much greater
than the number of samples, which, as you know, is generally considered to be undesirable.
We would much rather have a data matrix of size and by 100 than a data matrix of size and by 100000.
In addition, more dimensions simply means more data to process, which takes more time, and that's
also not good.
So reducing dimensionality is helpful because it means anything further in our pipeline will be more
efficient and fast.
OK, so what about visualization?
Well, visualization is always useful in data science.
We would like to see our data.
Notice how we do this all the time when we imagine how regression works or how classification works.
We think of them in two dimensional spaces.
It helps us to see how they work.
Of course, the world around us is only three dimensional, so it's not immediately obvious how we can
visualize a 10000 dimensional dataset.
The next thing to discuss is why reducing dimensionality and visualizing your data are actually almost
the same thing.
Well, suppose that our data has dimension at 10000.
This cannot be visualized.
But suppose that we could reduce dimensionality to transform it in such a way that our transformation
had less dimensions while still retaining all the important information.
This might sound like magic, but it is in fact possible.
Now, supposing that we could do this, then let's say we simply reduced dimensionality all the way
down from 10000 to two or three.
Then we could plot this data and look for any useful patterns and so forth.
Thus, reducing dimensionality is what allows us to visualize.
We can only visualize two or three dimensions and to get data in two or three dimensions, we must reduce
dimensionality.
So that's how these two ideas are connected.
Now, an obvious question to ask is, how do we actually know that data can be reduced in this way?
Consider what we do with Typekit modeling or even simpler clustering.
In the case of clustering, we assign each of our data points to a cluster.
A cluster is a set of data points where all the points are very similar.
Suppose that the data we used for clustering had 100 dimensions, but we chose to only have three clusters.
In this case, we would have reduced the dimensionality of our data set to three.
Now, obviously, this picture is just two dimensional because I want to visualize clustering, but
in actuality, this would be happening in 100 dimensions.
As we've discussed, you can't see a 100 dimensional data set unless it is first reduced.
Another example is with heights and weights.
We know that generally speaking, if you are taller, you are bigger and if you are bigger, then you
weigh more.
There seems to be a common factor here, which is size.
Imagine a line going through these points and that what we would like to do is project all of our height
and weight data points onto this line.
If we can do that, then we've reduced our dimensionality from two down to one.
Specifically, any point at the bottom left corner would have been small size in any point.
At the top right corner would mean large size.
But clearly, we don't need two numbers, which are height and weight to tell us about a person.
Instead, we can describe them with only one a number, which is size.
Now, when we started this lecture, I mentioned that there are three concepts we want to understand.
Visualization a dimension reduction and rotation.
We've looked at the first two, but not the third.
What I hope to convince you of next is that all of this is possible by rotating our data to understand
how this works.
We have to kind of work backwards.
We'll start with a story about how we think our data came to be.
You see, the whole point of dimensionality reduction is that we believe our data has a small number
of dimensions, but it is embedded into a larger dimensional space.
That is why it can be reduced in the first place.
If your data is truly of the same dimensionality of the space it lives in, then it cannot be reduced.
It can only be reduced if we believe that the data comes from a smaller dimensional source and is embedded
into a larger dimensional space.
To understand this, let's start with a simple, happy face, this happy face is clearly just a set
of data points in two dimensions.
And as a side note, if you're curious about how I generated this plot, I encourage you to try it as
an exercise.
OK, so now suppose that I took our happy face and embedded this two-dimensional object into a three
dimensional space.
This is exactly what we imagine happens with any other data set where we attempt to use dimensionality
reduction.
It's a smaller, dimensional set of data points embedded in a larger dimensional space.
Now, here's the issue.
When our two dimensional object gets embedded into this three dimensional space, it doesn't necessarily
align with the axes.
Therefore, it's not easy to see the happy face unless you are at a nice orientation.
And obviously, the ideal orientation would be to see the happy face head on.
But notice how some orientations can be very bad while others can be better.
For instance, if you picture the happy face facing upwards or to the side, you can't see the face
at all.
It just looks like a flat cloud of data points.
Of course, this is what happens when you embed a two dimensional object in a three dimensional space.
The two dimensional object is like a sheet of paper.
So when you look at the sheet from the side, you can't see the sheet.
For us, we can see a bit of thickness because they also added some noise in the third dimension.
Noise is an important aspect of CVD because we imagine that this process also ignores any dimensions
which are just a little bit of noise.
But consider what I'm doing here.
I'm rotating the graph, using my mouse, and under these various rotations, you can see the varying
degrees of usefulness of each perspective.
Some perspectives are useful since they allow us to see the face, and some perspectives are not useful
because all we can see is the noise.
So what is our goal?
Our goal clearly is to rotate our perspective such that we can see the face in the correct orientation.
That is, we would like to see the face head on facing us out of the screen.
And this is exactly what SVT does.
You can imagine that if we could find the exact orientation, we need to see the face.
The third dimension, which is just noise, would be coming out of the page.
At that point, we could simply discard this third dimension and keep only the two that are useful.
By doing this, we would have reduced the dimensionality from three down to two.
OK, so let's suppose we applied as we do to our happy face, which is currently embedded in 3D space.
Actually did this and this is the results you'll notice that the happy faces upside down.
This is perfectly OK.
Remember that the job of us video isn't to know that the eyes are at the top in the smile is underneath.
Obviously, there's no way for SVT to know the structure of a face, but what it does know is that the
third dimension was completely useless and that the true information in the data consisted of the two
dimensional happy face.
Now, because of the ambiguity of language, there may be some confusion about this demonstration,
in particular, I said that I added a third dimension, which is just noise.
So you might be wondering if I want the two dimensional happy face.
Why can't I just remove that third dimension, which I said is just noise?
And the reason you cannot do this is because of how we imagine our data is created.
Recall that our two dimensional happy face is not aligned with the axes.
It's at a random orientation.
But if we were to choose, say, Dimension one and Dimension two without first rotating the data correctly,
you would get a squished version of the face.
This is because the happy face was embedded randomly in the three dimensional space.
So it's equivalent to just picking a random perspective, which of course, is unlikely to be correct.
Mathematically, you can imagine our data points being an end by three matrix.
And as the number of data points and three is obviously the number of dimensions representing the three
axes.
So in the UN transformed or original space, you can imagine that each dimension contains both a bit
of data and a bit of noise.
Everything is mixed up.
If we choose only two of these dimensions to look at, we're still looking at a mixture of both a bit
of data and a bit of noise.
The goal of transforming this data or, in other words, finding the right rotation is to get back another
end by three matrix, which is organized more nicely, in particular in our new transformed and by three
matrix.
The first two dimensions are purely data.
While the third dimension is purely noise.
So that's what we're doing when we rotate our happy face.
Because the happy face is oriented or randomly, you have a bit of the happy face in all three dimensions.
But after rotating the happy face correctly, you move the happy face to the first to two dimensions
and you move the noise to the third dimension.
Now, in practice, there's going to be many more data dimensions and many more noise dimensions.
But hopefully this lecture helped to give you some intuition about what SVT does.