So in this lecture, we'll continue discussing LDA and specifically we'll talk about the intuition behind
how LDA works now.
LDA is a topic which could cover a whole course by itself, and it has many prerequisites in my experience,
more than typical deep learning and machine learning and so forth.
So this lecture will discuss some high level ideas, and if you want to know the details, check out
the paper in extra reading dot text.
Now you'll notice I mark this lecture as advanced.
Why is it advanced if it's just intuition?
The answer is that I'll be mentioning some of the techniques we use in Bayesian machine learning, which
should be intuitive if you have some experience, but we'll probably seem like magic if you do not.
This is just to give you some idea of what kind of prerequisites would be needed if you wanted a full
understanding of this method yourself.
Now, our goal in this lecture is to essentially understand this picture.
This is called a graphical model, which uses a convention called plate notation.
I want to start with this because although it looks complicated now, hopefully I'll be able to fill
in the blanks by the end of this lecture.
So just keep this picture in mind as our goal.
So to begin building our intuition, I'd like to start with clustering, but instead of just clustering,
let's compare it with classification, which is the supervised analog, as you recall.
These are both essentially doing the same thing, which is mapping an input X to some Category Z or
Y.
In the unsupervised case we call a Z, and it represents a cluster identity.
In the supervised case, we call it Y, and it represents a class.
But although they have different names, they both still represent discrete categories.
So you'll notice that for our classifier, we are using the Bayes classifier, which, as you recall,
is a generative approach.
In fact, both of these are generative models.
In the supervised case we have whi which points to X and in the unsupervised case we have Z, which
points to X. Now there's a slight difference in how I've colored these circles.
In the supervised case, both X and Y are dark.
But in the unsupervised case, only X is dark while Z is light.
This is not a mistake in graphical model notation, and dark means it's a variable we observe, while
light means it's a variable we did not observe.
But I want to point out another similarity.
If you have experience with these models, then perhaps you know that they both use Bayes rule to find
what is called a posterior distribution in the supervisor case, we want P of Y given X.
We do this by applying Bayes rule in particular p of X given y times B of Y divided by Pivac's Pivac's
given a Y describes the relationship or the arrow between X and Y, for example, given a Class Y X
is a Gaussian in the unsupervised case, we want P of Z given X.
Again, we do this by applying Bayes rule, which is p of X given z type of z divided by P of X.
So it's the exact same equation, except the Z replaces Y.
Now you realize that this poses a problem, as you recall.
Z is not observed.
Therefore, we don't actually know P of Z.
And since we never see Z, how can we compute any of this?
The answer is through a method called expectation maximization.
Of course, we won't discuss how that works in this course, since that's essentially a whole course
by itself.
But again, I want to let you know the names of these techniques.
In case you want to study them yourself, if you're interested in resources to learn these techniques,
just contact me on my website.
Now, using these graphical models, it allows us to tell a story about how our data was created.
This is where our model assumptions are embedded.
Since the models I've introduced are quite simple.
It might seem trivial, but I hope that this makes more sense as we encounter further examples.
Now being that both of these models have the same graphs, they both follow the same data generating
process.
Let's start with the super vice case.
In the supervisor case, suppose that we have two classes spam and not spam.
What this model tells us is that since there are no arrows going into the wide circle, these values
are simply generated out of thin air from of.
Why not given any other information, for example, of why might be 20 percent spam and 80 percent not
spam?
So using that distribution, we can sample a realization of why.
Suppose that for this instance, why is spam from this?
Why we can then follow the arrow to X?
Suppose that x seven y is multi nominal as we've seen.
Then we simply draw a sample from this multi normal distribution, which gives us, say, five instances
of car, six instances of insurance, three instances of cash and so forth.
Note that since this multi no is dependent on why there will be different distributions, depending
on whether why was spam or not spam.
So anyway, this will result in our X vector, which contains the various counts for the words in our
corpus.
Importantly, notice how this is an unrealistic process.
Nobody generates an email this way.
Instead, we generate emails by writing them so that they are coherent and makes sense.
We don't actually create documents by picking how many times each word should appear.
But remember that this is a model and models always come with simplifying assumptions.
It may not be realistic, but it does the job we need it to do in these instances.
I always like to remind students of the quote by famous statistician George Box.
He says all models are wrong, but some are useful.
So this obviously applies here.
Now, let's consider the unsupervised case as mentioned since the graphical model is the same.
There actually is no difference in the data generating process.
We still sample a z, which again comes out of thin air and doesn't depend on anything else.
Then, given Z, we follow the arrow to X, which tells us how to generate X given the different values
of Z.
The major difference in the supervised case and the unsupervised case is with the data we are given.
The model is the same, but the data is different.
We never get to observe the ZS, but we do observe the what the Z's are simply absent from our dataset,
which means we have to guess.
In addition to guessing what they mean.
You'll recall that we also have to guess how many there are in the first place.
Now, there's one small addition I want to make to our graphical model, which is to introduce the full
plate notation.
As you can see, I've had it rectangles two hour diagrams with the letter N. in the bottom right corner.
Basically, you can think of this like a for loop in the letter N. That tells you how many times that
loop runs.
So you can imagine this process and pseudo code.
We have a for loop with the index.
I going from one up to n inside the loop.
We first draw sample from P of Y to get a realization of Y, which we do know y survie.
From this, we then sample from P of X given y Saibai to get x of--i, which is a document belonging
to the Class Y Saibai.
So that's pretty much all there is to it.
So now that we understand clustering from a graphical model perspective, we can think of unsupervised
learning in sort of a big picture sense.
This slide shows three different unsupervised models in order of increasing complexity.
So the left side is the simplest and the right side is the most complex, which is LDA.
On the left, we have what is called a u anagram model, which will describe shortly.
We can visualize a MoneyGram model in terms of a single, multivariate Gaussian.
Although do note that we don't typically use Gaussians for word counts.
The point of this diagram is to show that there are no clusters.
Our model assumes that all data comes from the same cluster or, in other words, the same topic.
In the middle, we have clustering once again, as you may have learned in the past.
One way to view clustering is that it's a mixture model.
Again, we can visualize this.
So this is a Gaussian mixture model, but again, be aware that we typically do not use Gaussians for
NLP.
Finally, on the right side, we have LDA, which unfortunately is not as easy to visualize.
However, graphical models in played notation give us the language we need to describe it.
So the simplest model is the one where there are no hidden variables and no clusters.
We simply have the word by itself in the paper.
They call this the you anagram model, which makes sense since it's a bag of words and each word is
independent of the other words.
You'll notice that here we have two plates which are nested.
In particular, we have the outside plate, which is a loop over and samples.
And we have the inside plate, which is a loop over de words.
Of course, this results in an end by documents, by terms matrix, as we've seen many times using pseudocode.
We can again imagine the data generating process as a nested loop.
So we have four i from one to N 4G from one to B.
Then for each exigé draw a sample for the number of times where J appears in Document I.
Let's now return to the clustering model, which in the paper is called a mixture of anagrams.
In this picture, I've again used two nested plates with one over the samples and one over the words.
As you can see, for each sample, we first generate a topic Ze'evi.
So there's one topic per sample from this topic.
We then loop through each of our D words.
And for each of these words, we generate a count exigé given z sub by.
So at this point, we're finally ready to understand LDA.
The reason I've done things this way is because the model is kind of weird and how it generates topics,
whereas previously we had one word for topic with LDA.
We actually sample a new topic for every word that is, we sample a topic as many times as we sample
the word count.
You'll notice that the LDA model has many more variables in addition to Z, which is the topic, and
X, which is the word.
We also have theta, beta and alpha.
These are called priors, which are an integral part of Bayesian models.
We'll discuss that more later, but for now, we can use what we've learned to understand the data generating
process as you can see Alpha and beta show up outside the plates.
So these are fixed for all end documents and all the words inside the first plate.
We sampled Theta for each of the documents.
Theta is a prior for the topic z.
Well, Alpha is the prior four theta from Theta.
We follow the arrow to Z, but this is inside another plate.
This means that for each of the documents will be sampling a new topic d times, but notice that these
topics are not completely random because they depend on theta.
So practically speaking, you can imagine that Theta might be biased to produce topics in tech like
software, Microsoft, Spam, Google and so forth.
Now you'll notice that both the Xanax are inside the nested plates.
This means that they are both sampled the same number of times.
In other words, we can think of both Z and X as matrices indexed by AI and J.
Although X is a matrix of word counts and Z is a matrix of categories which are topics.
Again, I want to mention that this is weird because we normally think of a document as belonging to
one topic or a mixture of topics.
Llda is more complex because it samples a different topic for every word.
Finally, note that the word count X also depends on beta, which you can think of like a prior over
the word counts.
Now, this is essentially as far as I want to go in explaining this topic.
The key part of this was to understand plate notation so that you can understand the data generating
process, which means we understand the model assumptions of LDA.
We won't get into the finer points of this model like what it means to be Bayesian or how the model
parameters are estimated.
If you're interested, basically the key principle in Bayesian learning is that everything is a random
variable.
Because of this, everything has a distribution and every distribution has parameters.
When those parameters are priors to other parameters, we call them hyper parameters.
So, for example, under normal machine learning circumstances, Theta would be a parameter or, in
other words, a set of numbers or weights.
And that would be it.
But since this is a Bayesian model, this theta is not just the set of weights, but a distribution
over weights.
This distribution has a hyper parameter called alpha.
This also brings us to why this model is called latent directly allocation in the first place.
Now this goes far beyond this course.
But if you studied Bayesian machine learning with me in the past, then this should make sense.
As you recall, in Bayesian machine learning, we try to pick convenient distributions to work with
to make our computations simpler.
It turns out that for some distributions, there are special priors called conjugate priors, which
makes it very easy to apply Bayes rule to find the posterior.
Now, because their topics z are discrete, they come from a multi normal distribution.
It turns out that the conjugate prior for the multi gnomeo is the deer Ashleigh.
This is why Theta is sampled from a deer Ashleigh, which explains why this model has the name it has.
And as a side note, you can check all these facts on Wikipedia.
If you want to confirm that they are true.