So in this lecture, we will be summarizing this section of the course, which was on vector models
in NLP.
This section was essentially about how to make the connection between text, which is represented as
a string and numbers, which is what we required to do any kind of analytics.
Thus, this section was about how to convert text into vectors.
Part of this section was about text preprocessing, since this is a necessary step before conversion
into vectors can happen.
In particular, we learned about tokenization, which is the process of converting text into tokens.
We learned that tokens can be words, characters or sub words.
We also learned about the bag of words concepts where we ignore the ordering of the tokens in the document.
We also learn that not all words are equal.
Some words like and it and is can appear in any kind of document and thus are likely not informative.
We call these stop words.
We then learned about stemming in limited zation, which involves converting a word into its root.
This is useful because we don't want to have to independently learn about each variation of a word.
We already know that words like run runs, running and ran all mean the same thing.
Furthermore, this helps us to reduce the dimensionality of our vectors.
One simple reason we want to do this is because the more dimensions we have, the more time it takes
to do computation.
We then learned about several techniques for converting text into vectors.
The simplest being counts of tokens where each token is a separate vector component.
Using this method, we were able to build a text classifier with pretty good performance.
We then learned about TFI Taf, which handles one of the problems with count vectors, which is that
it doesn't account for non informative words that may have high counts but show up in many documents.
We also looked at the concept of Vector Similarity, which we applied to build a movie recommendation
system.
As an advanced exercise, we then looked at how to implement TFI Taf from scratch.
This helped to give us deeper insight on how TFI Taf actually works.
Furthermore, it introduced us to an important concept, which was the word to index mapping.
This is especially useful in deep learning where we work with the indices and not the words themselves.
The final topic we looked at was neural word embeddings, which are word vectors used in deep learning
and neural networks.
Specifically, we did a little preview of words of and glove, which, unlike the other methods we've
discussed, convert words into vectors instead of hold documents.
We then saw how these vectors can do interesting things like word analogies.
They showed us that the simple process of converting words into vectors does something very meaningful.
Words really represent concepts on multiple dimensions.
For example, take the word king.
King represents the ruler of a society in one dimension.
King represents mail in another dimension.
King is a noun in another dimension.
In fact, these dimensions are not abstract, but rather numeric.
This is actually a very interesting idea, since if we can represent concepts with numbers, then it
may also be the case that thinking and reasoning are simply mathematical operations on those numbers.