In this lesson we will talk about what are tokens?
In my opinion, the best way to define tokens is saying that they are the atoms of the LMS.
So it's like the minimum unit to measure.
What do we do with LMS?
A token is typically a small part of a word.
In some cases, one complete word is a token.
A token, but in many cases a token is typically a small part of a word.
So if you remember, we said that ChatGPT has been trained with the text equivalent to 10 million books.
So if we.
Take a look at the number of words in 10 million books.
We are going to have around 800,000 million words.
So the equivalent in tokens is a little bit more than that is actually 1,100,000,000 tokens.
Okay.
So.
Yeah, we would say that.
1.5 tokens equals to one word, something like that, more or less.
So you have also other kinds of a names.
In LMS, multiple tokens form a sequence and many sequence form a vocabulary.
But in our case we are going to stay with tokens.
We don't want to, uh, worry ourselves with sequence or vocabularies.
Tokens is going to be enough for us a in building LMS applications.
In the next lesson we are going to talk about prompts.
This is an important topic.