So in this lecture, we'll be discussing the intuition behind lane directly allocation, also known
as LDA.
We'll start this lecture with a simple outline of what will be discussed.
The first thing I'll point you to is the paper that introduced this model.
Since today, it's considered a classic.
And a lot of the techniques should be familiar if you have some experience with Bayesian methods.
The next step in this lecture will be to look at unsupervised learning at a very high level.
This will help you understand how it contrasts with the supervised methods we looked at earlier in this
course.
We'll spend some time looking at different kinds of unsupervised methods, such as clustering and how
that fits into this picture.
In a sense, clustering methods could also be used for topic modeling, since each cluster would be
a separate topic.
For example, Document one might belong to Cluster two, while Document two belongs to Cluster three
and so forth.
But you'll see how LDA it gives us some more rich description of clusters.
Once we finished discussing unsupervised methods at a very high level, we'll start looking at elders
specifically.
We'll try to understand it from what I call an API point of view.
In particular, we'll ask questions like What are the inputs, what are the outputs and how can I interpret
those outputs?
We'll see how this translates to how we use LDA in Python.
Since LDA is part of CI get learn, the inputs and outputs are really what we'll have to deal with.
This is the first step towards understanding what LDA does.
The final part of this lecture will look at the intuition behind LDA.
LDA is a Bayesian model, which admittedly probably means nothing to you if you don't have experience
with it.
But I'll try to give you some appreciation for what this entails Bayesian models or graphical models,
which allow us to impose some structure and assumptions on the data generating process.
As a quick side note, please be aware that Llda is also an acronym for another machine learning method
called linear discriminant analysis in older versions of and learn.
There's a model called LDA, which refers to this and not the topic of this lecture in newer versions.
This was moved to a module called discriminant analysis so as to avoid confusion.
OK, so as promised, the first topic of this lecture is just a quick mention of the paper, which was
authored by David Blye, Andrew Ng and Michael Jordan, all of whom are prominent researchers in machine
learning.
I'm sure you've heard of at least one, if not all three.
Now the reason I want to mention this paper is twofold.
Firstly, note that these lectures will only discuss some basic intuitions about how LDA works.
Actually learning all the details behind LDA is a significant task, and it would be very advanced if
you are an advanced student and you want to know these details.
The paper would be a good next step after watching these lectures.
It's included an extra reading in the course repo so you can find it in the usual place.
The second reason I want to mention this paper is because it's a beautiful demonstration of just how
far you can go on machine learning, and it uses a lot of interesting advanced techniques.
These include probabilistic graphical models, expectation maximization and variational inference.
All of these are important topics in Bayesian machine learning.
Even if you do not understand the techniques being discussed, you should take some time to marvel at
the complexity behind this method.
In fact, one more interesting note to make about this paper is that it begins by building off from
latent semantic indexing, which is a topic we covered in this course.
So it forms a nice connection with other topics you learn about in this class.
OK, so now let's answer the question.
What is unsupervised learning and why is it useful?
So the best way to understand unsupervised learning is the first thing about supervised learning.
Let's consider a simple task where a job is to predict given a piece of text, whether it's about physics,
biology or math, in order to build a model to perform this task.
We need a labeled data set, as you recall.
You can think of this like an Excel spreadsheet with two columns.
In this case, one column is the input, namely, the text in the second column is the target, specifying
either physics, biology or math.
The difference between supervised and unsupervised learning is that with unsupervised learning, we
are only given the text, but no labels.
As always, you can visualize machine learning by plotting your input vectors on a grid in the supervise
case, each data point is colored and those colors represent the classes in the unsupervised case,
the data points all have the same color because we're not given any information other than is simply
where those points live.
Our job, then, is to figure out what we can do even when we're not given these colors.
So what if, given this tax, we would still like to categorize these documents?
Is this possible even if we are not given any labels?
Surprisingly, the answer is yes.
We call this clustering in the canonical example is K means, which you've probably seen in the past.
Now, as you recall for machine learning, our first step is typically to convert our input data into
vectors, which you can visualize like points in a grid.
This will continue to be the case in these lectures.
So let's suppose we've converted our text into vectors using some method like counting or TFI Taf,
and we then cluster these vectors like so.
One important fact about unsupervised learning is that these clusters have no meaning by themselves.
We might no these clusters, for example, we might call them clusters one, two and three.
But this is unlike supervised learning where we know what each category means.
For example, physics, biology and math with unsupervised learning, there's no guarantee that our
three clusters correspond to these three subjects, even if we trained on a physics, biology and math
dataset.
The only way they attain meaning is by what we assign to them based on what we know.
Another important fact to remember is that we don't even know how many clusters there should be.
Occasionally we can simply look at the data if it's two dimensional and very simple, or we can try
to guess based on some out of sample metric.
But generally, the number of clusters is unknown and it must be chosen by us.
This is also unlike supervised learning where we know the categories because they come with our data
set and because we're given these categories, we know exactly how many of them exist to foreshadow
a little bit.
This also means that for topic modeling, we don't know how many topics there should be.
We simply pick a number and see how it goes.
Now, at this point, it's useful to turn our attention to the practical side, which is just to look
at LDA in terms of an API, that is what are the inputs to this model and what are the outputs?
This will help you understand what we previously discussed, and it will help you understand what happens
in the code example.
If you don't like math, then this is essentially as far as you need to go.
However, if you do want to know a little bit about how this model works, then that will come after.
OK, so the inputs to this model are quite simple.
Just the count vectors we learned about earlier in this course.
Note that this is a bag of words model, and so we don't take the order of words into account.
As you recall, this can be done using the count.
Victories are inside it.
Learn to review how this works.
Each document is converted into a vector, and each element of the vector represents a word count since
we have multiple documents in our corpus.
Our dataset will then be a matrix, with number of rows equal to the number of documents and columns
equal to the number of terms.
So the outputs of LDA are as follows, we essentially get two matrices as output, although I'll qualify
this statement later.
The first matrix is a matrix of topics by words.
The second matrix is a matrix of documents by topics.
So how can we interpret what this means?
Well, it helps to simply look at a picture.
Let's start with the topics by words matrix.
For this matrix, each row is a topic, and each topic is represented by a vector, which is that row.
Obviously, the number of components in the vector is the number of words in our corpus.
Now, as you recall, for Elder, each topic is a distribution over words.
Or put another way, not a discrete category.
So as you can see, one way to visualize the topics is to simply plot these distributions as a bar chart
showing the top words for each topic.
That is the words with the highest probability.
In other words, these are sorted.
So in the first chart, we can see that topic one has the highest values for words like software, Microsoft
virus security and so forth.
Clearly, this topic has something to do with desktop software.
And notice that what I've just done is an example of how we assign a meaning to topics.
Let's now look a topic to the top words are people government, US, UK human rights law.
This topic appears to be related to politics, and we can do the same thing with the other topics as
well.
Now, let's look at the second output matrix, which is documents by topics again, we can visualize
each document with a bar chart displaying the amounts each topic belongs to that document.
Notice how this is like a soft clustering, since we don't assign a document to just one topic.
And this actually makes a lot of sense, since documents can be about multiple different topics, depending
on how those topics are differentiated.
For example, we previously saw two topics related to software, but with different words.
In one case, it was centered around Microsoft and viruses and security.
In the other case, it was centered around Google and search and spam.
Well, what if we had a document about how viruses were acquired through spam?
Now it's a document that has components of two different topics, so that should give you an idea of
why probabilistic topic assignments are useful.