-: All right, let's talk Prompt Optimization.
So we're gonna define a couple of prompts,
and we're going to A/B test them
and see how well they do.
And this is really key, I think,
when you start talking about prompt engineering.
You're really talking about actually optimizing your prompt
and proving that it works in production,
finding all those edge cases.
Because you don't always get that
when you're just playing around in ChatGPT, right?
So, I think this is really the next level
that you can start to get to.
And once you start to learn
about the benefits of prompt optimization,
I think then, you start to really unlock
a whole new kind of tier of performance.
So let me just run this.
(mouse clicks)
What that's gonna do is it's gonna use getpass
to help me just paste in my API key
without, you know, actually just sharing it with the world.
So I recommend using that.
And then that's stored locally.
So I'm just gonna run this,
which is to install OpenAI,
and that's the only library we really need.
And we're gonna be using the openai.ChatGPTCompletion.
So we're gonna actually be calling GPT-3.5 Turbo
to test this.
Typically, I would say prompt optimization
is really important for smaller models,
like, if you're using GPT-4,
it tends to work a lot better, but that's very expensive.
And if you can get the prompt to work on GPT-3.5,
then you're gonna save like a hundred, you know,
like 99% of your costs.
So now, these are the two prompts that we wanted to test.
And the reason why we want to test them,
so this is, you know, product name generator,
and you know, if we're gonna be doing this
hundreds of times, you know, maybe we're building a product
that helps you generate product names, kind of matter,
then it makes a big difference in terms of token costs
and also, whether you can get it working with 3.5.
So prompt B and prompt A are the same.
The only difference is that prompt B has two examples.
So it's a few shot versus a zero shot prompt.
And we're gonna actually test and see
if adding those examples really helps.
And the reason why that's important is,
if we don't need to have them in there,
let's get rid of them.
I see a lot of this, like, prompt witchcraft, right?
Spell casting, some people call it blind prompting,
is how I've heard it described,
where you don't actually really test the prompt,
you just keep adding more and more stuff
until it becomes a big essay.
And I would bet you that most of that stuff isn't necessary,
it isn't followed and maybe even leads to worse results.
So it's really important to test.
All right, so we're gonna test these two prompts.
We're gonna see if adding the examples really helps,
and we could test other parts of this, right?
We could test whether it works well for multiple seed words.
It doesn't have to be performance.
So it could be, you know, do we get bad language?
You know, if we put like a swear word and the C words,
like, you can test this for all sorts of things.
So lemme show the interface.
So we just get the prompts together in this test prompts,
we can actually add multiple prompts if we wanted to.
And we're just gonna import Pandas
and OpenAI Pandas is the data frame.
We're going to store all the tests in
and then do some reporting.
So this is just the standard way that you talk to GPT-3.5.
I'm just using the standard, you know, system message.
You could obviously use something different.
And then, we're just returning the actual text here.
So, we have the test prompts, we have the responses,
so we're gonna save them.
We're gonna run five tests per test prompt,
and then, we're gonna evaluate them afterwards.
So this is what runs the test.
And then, we're gonna store them in the data frame,
and we're gonna print the df,
we're gonna save it to responses.csv.
So let me just run this.
(mouse clicks)
And what this is doing is calling OpenAI multiple times.
(clears throat)
So it's gonna just take a few minutes.
It's calling it 10 times for each prompt.
Should wait for that to finish.
Should only take a few more seconds, I think.
One thing I should point out is,
every time we're calling this, saving the variant name,
the prompt, and the response we got.
And the variant name, we're just calling it, the first one A
and then, this will find B for the next one,
C for the next one.
Here we go.
So, we have all the different responses, which is cool.
Oh, one thing I would say as well,
if you're doing this for lots of prompts,
so many, many different, you know, calls,
then, you might want to add some retry logic
in here as well.
All right. So now, we're gonna read the CSV,
responses.csv.
Actually, we can see it, it's in here.
Yeah, responses.csv. So we can grab that.
And then, we're just gonna shuffle the index.
So it's really important when we evaluate the responses
that we're, you know, doing it blind,
like, we don't know which prompt it came from,
so we didn't skew the results,
but also that they're shuffled.
So again, we can't, you know,
we didn't know which one it came from.
Otherwise, you know, you can see that like,
these all came from the first prompt
and like, these all came from the second.
So, you know, it'd be too easy, otherwise.
All right. So what does this code do?
This just sets up an interface where we can quickly
click thumbs up, thumbs down,
and it just uses the IPython widgets.
So this only works in a Jupyter Notebook like Google Colab.
So what it's doing is, it gives user feedback of one,
if there's a thumbs up and zero if not.
And then, we are updating the response to load the next,
you know, response from the CSV.
And then, if we've run out of responses,
then, we save results.csv,
and then, we do some printing of the feedback
and kind of give us the the mean.
All right, so this is the update response.
This basically just gets the next thing in the index,
the next response until we're all done.
And then, it just does some formatting to kind of show you,
you know, this in the interface.
So, this is all setting up,
like, the thumbs up button and all that stuff.
So, don't worry too much about that.
So we're gonna run this. Here we go.
So we have it. So Adapt-a-fit, OmniShoe, FitAll, FitFlex.
So I think that's good.
What I'm gonna test for here
is whether it's in the right format.
So, forget about the quality of the names right now.
I just wanna make sure, are we getting the right format?
And this is the format I want, right?
I want a inline comma separated list. So that one's good.
This one's the same. That's good.
Okay, that's different names,
but it's still in the right format.
Okay, now this is wrong.
Here we go. We have a failure. (chuckles)
Perfect.
So, it's a numbered list in order,
and that doesn't make any sense.
And if we were trying to pass this,
that would really suck, right?
Like, when we're trying to use this programmatically,
then sometimes, it comes through like this, you know,
it's not gonna pass properly.
So you say no and then no for this one,
and then no for this one, no for this one.
Okay, this one's good. This one's good. No for this one.
So it looks like sometimes it's coming through.
Here we go. Yeah, look, prompt B, score of a 100%.
Prompt A, score of 0%, right?
So that's a real problem, right?
Like by, you know, not including the examples,
it's not following the, you know,
the format that we want, right?
So it does look like we need those examples in there
to get the right format.
And yeah, we wouldn't have known that if we didn't test it.
I didn't think it was gonna be a 100% right.
It's a real problem.
Cool. All right.
But now that you know how this works,
like, you could really test it for everything.
You know, you could test it for quality,
you could actually add ratings if you want in here
instead of just thumbs up and thumbs down.
You know, the world is your oyster.
And hopefully, you'll be testing
all of your different elements or your prompts, you know,
instead of just taking on blind faith that it's necessary
or even helping.