-: Right, welcome back.
So in this video, you're gonna have a look
at how you can count tokens
when you're using the OpenAI API,
and we're gonna be using a package called tiktokens.
In your GitHub repository, I want you to open
the how_to_count_tokens_using_tiktoken notebook.
And we're gonna just go through
and run a couple of the cells
and be familiar with understanding
what this Python package is.
And there's a lot more information if you wanna go
into the depths of how tiktoken actually works.
But essentially, tiktoken is a Python package
that we can use and it is a tokenizer
that will take those larger pieces of sentences
and documents and convert those into tokens.
The first thing you're gonna need to do is you're going
to need to install tiktoken and OpenAI.
And then what we're gonna do
is we're gonna import the tiktoken package.
After you've imported tiktoken,
then you can use a specific type of encoding.
And there's different types of encoding
for different types of models.
The one that you'll find is often used is the cl100k_base.
You can then use the tiktoken.encoding_for_model,
and if you'll see here, you're then able
to do a .encode
and you'll see how many tokens this piece
of text becomes out for gpt-4o-mini.
So we end up having 1, 2, 3, 4, 5, 6 tokens
for that tiktoken is great.
So you can use this tiktoken.encoding_for_model
and then giving it the model name.
And then you will use the .encode method
to figure out exactly what those tokens are.
You can also then have a helper function
where you take in a string and an encoding_name
and you do something like this,
and then you're just counting the length for those
that get returned from that .encode.
So you can see here, we're gonna just call this here
and you've got around about six tokens.
You can also turn tokens back into words
by decoding them using that same encoder.
And as well as that, you can also get the total number
of single token bytes.
And you can see here what that specific token is
after the b character.
We can look at different types of encoding
and you can see this is how you could compare encodings.
We're gonna now look
at how you could count a ChatGPT completion call
using a helper function called num_tokens_from_messages
where we pass in some messages, we pass in a model name,
we try and get that encoding_for_model,
and if there is an error, we just get a zero,
an o200k_base.
And then we look and see if there's models.
And then we basically use
the num_tokens_from_messages function
to calculate the different tokens there are.
We also add on some additional tokens
because every reply is primed with this:
<|start|>assistant<|message|> at the front.
Then what I want you to do is import OpenAI
and you're gonna need to set up your API key here.
If you scroll down,
okay, so we have a bunch of example messages,
one system message, then some user and assistant messages
and user and assistant and finally finishing with the user.
Then what we're gonna do is loop over all of these models
and we're gonna print the num_messages.
We're gonna print the output that we get back
from this num_tokens_from_messages function.
We're then gonna run a client.chat.completions.create,
passing in the model=model
and the messages=example_messages.
And we're then also going
to print out the .usage.prompt_tokens as well.
So that'll be the number of tokens
that OpenAI has calculated for usage.
And you can see here these token estimations match up
with what we were actually charged by OpenAI.
So this can be a really good way if you're hitting
into token limits or you need to know exactly
how to count the number of tokens.
Then you can use the tiktoken package,
which has a built-in encoder allowing you
to get out the number of tokens
for your OpenAI messages.
It's also possible to do more advanced things,
like calculate the number of tool calls.
So if you are using agents and tools,
feel free to have a look at this specific section.
To keep things simple,
we're gonna move along for now, but have a look at that
if you're really interested
in how you can calculate tool token costs for your agents.
The next thing that we're gonna look at
is how we can manage the message history if we want
to only have a certain number
of tokens using the tiktoken package.