-: All right, welcome back.
So in this video, we're gonna have a look at rate limits
and retrying within the Responses API.
I want you to go and open the notebook
rate_limits_and_retrying_with_tenacity,
and we have a variety of different ways
that you can get around errors.
So if you're doing lots and lots of API requests,
you might start running into something
called a rate limit error.
I'm gonna show you a couple of different ways
that you can overcome rate limits by using retrying.
The first thing we're gonna do is install some packages,
so openai, tenacity, and backoff.
Tenacity and backoff are custom rate-limiting packages.
You're then gonna initialize your client,
and you'll need to put your API key here
alongside with your model.
You can customize the OpenAI client.
By default, it will do two retries,
but you can customize this to do more retries.
So if you are hitting limits when you're using OpenAI,
you can increase the number of retries.
I will say that this will increase the amount
of time it will take
for OpenAI to come back with the output
if it's running into a rate limit error.
So that's one way you could do this
is customizing that max retries on the client itself.
Now you're probably wondering
what are the different types of rate limits?
Well, there's actually several.
There's the request per minute rate limits.
That's how many requests you can make
against OpenAI per minute.
There's also something called tokens per minute,
and this is where the fact
that you now know about tick token,
how to count tokens,
allows you to roughly work out
how many tokens a message is going to use.
As well as that, we can also make a request
against the endpoint.
And when we make a request with the request package
and we're just putting in one token
and we're just using the role of user in this content,
and then what you'll see
is we get back this response headers
and you'll see that it shows us
how many remaining requests we have.
If we go and have a look at the response.headers,
you'll see there's a variety of different things in here.
So the important ones
will be the version of API that you're using.
These ones, there's x-ratelimit requests,
so how many requests you have,
how many token limits you have,
and also things like the remaining number of requests
and the remaining number of tokens
and how long it will take until both the requests are reset
and also the tokens are reset.
You can also customize retries with the packet itself,
with the client package itself, with this .with_options.
And you can see here we're adding a custom max retries
on the chat.completions functionality.
You can also use a Python package called Tenacity,
and that basically will decorate a function.
So you can see we've just wrapped
the client.responses.create and all the kwargs,
and we've done it create_with_backoff.
Then we can pass in all the original arguments.
So the model is equal to the MODEL,
the input is equal to this "role": "user",
"content": "Tell me a joke".
And then that's basically setting up six retries
and it's also adding in some wait time
between those retries.
You can also code up a manual exponential backoff.
So if you really need to go quite low level,
have a look at this
where you're basically creating a function,
the initial delay, the factor, some random jitter,
and the maximum number of retries.
And then we are using function decorators
to basically loop over that function.
And you'll see here if there is an OpenAI rate error,
then we're basically going and sleeping
and then adjusting the delay.
And then you can use that like this,
so you've got this retry backoff,
you've got a create_manual,
and then we're just using that client.responses.create.
So there's another way for you to write that
if you wanted to customize the retrying a lot.
Now, my personal opinion on this
would be to customize the client
and how many max retries you set for the client.
Also, remember to visit your OpenAI platform,
look at the usage,
and then have a look at the various things.
So like the input tokens you're using daily,
the output tokens you're using,
and how many requests you're making,
that will give you a good amount of knowledge
about how close you are to the rate limits.
Now, where do you find the rate limits?
Click on the cog on Settings,
go to Limits.
And depending upon your tier,
you'll see you get different limits.
So for example, I'm on Usage Tier 1,
and these are my token limits per model
and my request per minute models.
Now, depending upon when you upgrade
to a different higher threshold,
basically if you spend more than a certain amount of money,
you will automatically upgrade
and you can actually buy credits
to make sure that your organization
goes to a new usage tier.
So remember, you've got two types of rate limits,
tokens per minute, TPM,
and requests per minute, RPM.
You need to respect both these limits.
The final thing I'll say
is if you're doing heavy, large workloads,
so you need to process 500,000 requests,
you should look at using the batch API endpoint,
which will not affect your daily usage quota.
The batch API endpoint, these rate limits are separate.
So you can run 900,000 tokens per day
on GPT-4.1
or 2 million tokens per day on GPT-4.1-mini.
You can still then use 500 requests per minute
and 200,000 tokens per minute for your daily usage.
So check out the batch API,
and it has its own separate limits for that.