So in this lecture, we will be discussing some basic definitions we'll be using in this course.
There will be more along the way as needed, but this is just to get us started with the basics as a
side note.
This course will be focused on the English language, so it would be best if you have familiarity with
it, which I'm sure is probably the case since the lectures are also in English.
So that being said, let's recall that we communicate in sentences.
A sentence is a sequence of words which typically begins with a capitalized word and ends with punctuation
like a period or a question mark.
OK, so hopefully you are familiar with all the terms I just used, specifically sentences, sequences,
words, capitalization and punctuation.
If you don't know what any of these words mean, let me know on the Q&A.
OK, so let's go through some more terms.
We've stated that a sentence is a sequence of words.
Sometimes we refer to words as tokens.
The term token is a more general term, which can refer to words, but also can refer to punctuation
or even sub units of words.
The details of this are best left to another lecture, but for now, just note that it is common to
use the term a token in place of word.
OK, so here's another source of ambiguity.
We know that words are made up of letters, as I'm sure you're aware the English language is made up
of the letters A to Z.
And there are 26 such letters.
However, like the term token, there is a more general term that we can use, which is the character.
The concept of characters is more general because they can represent not only letters, but punctuation
and spaces as well.
So, for example, a space is considered a character, as is a new line.
If you've ever coded in C++ or Java, then you should be familiar with characters as the character is
a data type in these languages.
Sometimes in NLP, we build models in terms of words or tokens, but other times we build models in
terms of characters.
Both have their pros and cons, so it's good to be aware of these possibilities.
OK, so let's go through some more definitions.
We know that different words make up a language.
What is the set of all words?
We call this the vocabulary.
Note that in most cases, our vocabulary will not consist of every possible word in the English language,
but only a reasonable subset of those words.
Sometimes this might number in the thousands, tens of thousands, hundreds of thousands, or perhaps
even millions.
The choice of how many words to use is really up to you.
You should choose this based on the results you get.
On the other hand, if you're using a pre-trained model, then you have no choice.
Simply use what is used by the authors.
The next term I want to introduce you to is Corpus Christi.
So here are two definitions I found by searching this term on DuckDuckGo.
The first definition is a large collection of writings of a specific kind or on a specific subject.
And the second definition is a collection of writings or recorded remarks used for linguistic analysis.
So either of these is fine.
Basically, when I say corpus in this class, it refers to the dataset, which we will be using to train
a machine learning model.
OK, so the next time I want to introduce you to is the Engram.
Basically, this just means a sequence of end consecutive items or tokens, whether they be words,
characters or sub words.
This is not a complex topic by any means, and in fact, we don't even really need to give it a name,
in my opinion.
However, this may help you if you're reading the literature and you come across this term.
Note that we have special names for small values of that.
For example, if we're looking at just single tokens, we call those you anagrams.
If we're looking at sets of two consecutive tokens, we call those by grams.
If we're looking at three, we call them tri grams.
OK, so you get the idea.
Now, where does this come into play?
Well, one instance you'll see this idea pop up is of Divac, which is basically just a fancier way
of building a neural network for by grams.
Another point of which you'll see this concept is with Markov models, which look at bigram probabilities.
Now again, it's my opinion that the terminology itself doesn't matter too much.
But if you see this term, then at least you know what it means.