-: In this video, you're gonna learn about
what is prompt caching.
We're gonna look at the prompt caching methods
for both OpenAI and Anthropic
and how you can understand these
to make decisions about your LM calls.
So, prompt caching was released by Anthropic before OpenAI.
They released us on the 14th of August 2024.
They have basically made a more cost effective way
for your input tokens
if you're reusing the same input tokens.
OpenAI followed suit on the following month in October,
the first of 2024,
and they introduced prompt caching in the API layer.
There is some major differences.
We're gonna explore what prompt caching is
and how you can leverage it as well.
So prompt caching is a way for you
to reuse frequent context in API calls,
and it's solely for input tokens
and doesn't affect output token prices.
So if you're always using the same inputs again and again
and again, then prompt caching is gonna save you money,
whether you're using OpenAI or Anthropic.
The common use cases that you'll use for prompt caching
is where you want to reduce the cost and the latency.
You want to use it for the same context again and again,
and it's ideal for long form content and conversations.
OpenAI offer prompt caching automatically,
so you don't actually have to do anything,
but apart from you have to use a prompt,
which has to contain at least 1,024 tokens,
and they will cache the largest thing,
which will be at least 1,024 tokens,
and it will lead to a 50% cost reduction.
The cache also expires
after five to 10 minutes of inactivity,
and it's available for currently GPT-4o, GPT-4o Mini,
o1 Preview, and o1 Mini.
If we have a look at Anthropic,
it's a little bit different in the sense
that the cost reduction is much greater
and also provides a latency reduction.
Now the pricing structure is based on both the cache write
and the cache read
so initially is slightly more expensive,
about 25% more expensive than the base price
of those input tokens.
However, when you're reading from the cache,
then it's only gonna be specifically 10% of the base price,
and it's available for haiku, sonnet, and opus models.
Now we're gonna have a look
and see in a notebook in the next video exactly
how you can explore prompt caching
in both of these providers.
You'll also learn how to calculate
the input token savings as well.