-: Okay, using DSPy is quite good for optimizing prompts,
but typically, you need
to set up your own evaluation framework first,
and that's actually the hardest thing,
not writing the prompts, but designing the evaluator,
particularly for tasks where it's not as straightforward,
you can't just calculate.
You need to actually create some sort
of synthetic evaluation
where the AI will tell you whether the joke is funny or not
because you can't use DSPy as a manual human being.
(chuckles) You have to have a labeled test set.
So what I did, this is a approach I use quite often,
this is just a list of funny jokes scraped
from the internet, and then I just asked ChatGPT
to make a list of not funny jokes.
Here's an example.
And so we have funny jokes and not funny jokes,
and then I just create a test set.
So if you see here, don't worry too much about this code,
but basically, it just creates a DataFrame
and this here is that to_csv for each one.
We grab all of the jokes,
we zip them together, and then we shuffle them
and then split them into test,
into training data, 177 results,
testing data, so that's what we're using
to optimize with DSPy,
and then development data, that's how we're checking
that DSPy hasn't over optimized on the test set,
and that the actual solution is more generalizable
to things it hasn't seen before.
Cool, and you can see that each one has got a topic
and a joke.
Now, one thing that's interesting,
by the way, is these initial ones here.
When I downloaded them off internet,
they didn't have a topic, obviously.
What I did is I just asked ChatGPT
to create the topic for each joke.
It's a way to, you know, if you have specific inputs
that you need, you can just reverse engineer them
from the outputs.
It's another little trick, (chuckles) which is quite fun.
All right, now, if we go through,
and now we have the funny jokes labeled as 1
and they not funny jokes labeled as 0.
We can create, we can set up our DSPy,
and there's some involvement here,
so I have GPT-3.5 Turbo and GPT-4 in case I need it,
and what I'm gonna use is GPT-4
to teach GPT-3.5 how to do this task.
Because if we can get GPT-3.5 checking
whether the joke is funny or not,
then that's much cheaper when we're actually optimizing,
because every time we optimize or test a joke,
we're gonna have to run this,
and we don't want to have to rely on GPT-4.
So in DSPy, you create a class, a signature,
and we just show that we have the joke and we have the topic
and we have a question to assess the joke against,
and then the answer, and in this case,
the question is just, "Would this joke actually be funny
"to an adult attending a comedy show?"
I found that prompt is what works best
to get a reliable result.
So that's my bit of prompting.
And then we can just test it,
so we feed in a topic and a joke
and then we have an assessing the joke chain,
and it gives you rationale, "need to consider the joke
"about heavy drinking and lighting candles
"on a birthday cake would be appropriate and humorous,"
blah, blah, and then here, it said no,
so it didn't do a good job.
I actually think this joke is funny
but it doesn't think it's funny.
And so we create a metric,
and then we can evaluate based on the existing dev set,
so that, remember, if we looked at it before,
we had training data, we had test data,
and then we had development data,
so development is basically our fair test
of whether it's doing well or not.
And if we look, it gets 42% of them correct.
And you can see here that it answered yes,
that joke is true, and then it is was actually true.
It said yes, but it actually, it was false.
It's making a lot of errors.
It's only got it 46% correct.
So how do we train it?
Here, we're doing bootstrap with random search,
and I'll just explain what these mean,
so we're loading in the training data set
and the testing data set, it's gonna use them to optimize,
and then we're just giving it a few parameters here.
We're giving it the metric, the evaluation metric,
and then, that was the prompt that we came up with earlier,
and then we're telling it how many bootstrap demos,
this is how many synthetic examples it will make up,
so GPT-4 will make up a good example of this joke
and then give it to GPT-3.5 to put it into its prompt,
and so we give it up to eight jokes
that GPT-4 has written that pass the evaluation metric.
And then we have labeled demos,
so this is how many prompts
from our training data can we put into the,
how many jokes from our training data we can put
into the prompt, so there's 16 here.
These are just values that I found
that have worked pretty well.
You'll get different results
for different tasks if you set different amounts.
Then, we have teacher settings here,
this is where we tell it we wanna use GPT-4 Turbo
as the teacher, rather than GPT-3.5.
And then we have number of candidate programs, 16,
and this is basically how many of the accommodations
to track during the random search.
So if we run this, normally it takes really long,
maybe 15 minutes, 10 minutes, something like that,
but this should have been cached locally,
so hopefully it doesn't take very long.
It looks like it's actually running it again
for some reason.
Maybe I accidentally deleted my cache. (chuckles)
Every time you run this,
it's gonna take a few hundred examples.
You can see there's 177 calls right now that it's doing,
and it's gonna give you the average metric.
So I'm just gonna pause this recording
and wait for it and then I'll come back.
Mm-kay, this is still running.
That's taking some time. (chuckles)
But you can see, in some cases,
it's getting a good response.
I'm just gonna keep going through this
and we'll come back to that,
but essentially, once you have this compiled
and it's found the best combination,
then you can run it like this,
so you give it the joke
and you can see actually, in this case,
from a previous run, it did actually check,
it confirmed that the joke was good,
so that's really helpful.
And you can run the evaluate based on the full program.
You can see how, on average, it got it,
and you can see here that the success rate was 79.5%
so it was up considerably from the 46% we had before.
Was, you know, basically almost double the accuracy
in terms of getting it correct, which is really helpful.
But I wanted to show you another option here,
which is called MIPRO,
and what this allows you to do is not just add good examples
to the dataset, but also come up
with new prompt instructions.
So here, the prompt model is gonna be GPT-4,
and then the task model's gonna be GPT-5,
it's gonna come up with new prompts for you,
as well as add different prompts to the,
sorry, different examples of jokes to the prompt,
and here we've gotta set the same results.
Should get the same results,
but it's also gonna optimize the prompt for us.
And this actually takes a lot of calls so you can,
I'm not gonna run this again,
it's 5,000 calls for this one, and so there's a lot there,
but once I ran it, then you can get an idea.
It's just printing out the different prompts,
so I'm gonna zoom through those.
You can see like how it makes its decisions in there.
But the final score, which is really interesting, is 84%,
so we got another few percentage points difference.
Tuning the prompt did make a bit of a difference,
whereas just adding the examples, I think,
made the most difference.
The takeaway I would say
in general is that adding good examples
to the prompt is a much bigger gain.
But to have a GPT-3 level prompt
that can generate 84% accuracy
in telling whether a joke is funny or not is really good
because now we can use that, you know, going forward.
And you can see, when you run this on the dev set,
you can still see, it's 84%.
This is based on the training set,
this is based on the dev set,
and we're still doing a really good job.
It didn't over fit.
If you wanted to see the prompt that it came up with,
this is what it came up with,
saying, "Follow the following format, the joke," et cetera,
and then, "Evaluate the humor of each joke
"by considering its sophistication,
"cultural references, and wordplay."
Yeah, so you could take this prompt to basically,
you don't have to use DSPy anymore,
you could take this prompt
and then just, it's just text, right?
So you could save it somewhere or put it somewhere else.
And here, I've saved this program as JSON,
now you have this prompt,
you could go and use this in a custom GPT or whatever,
or you could use this as an evaluation metric.
Enjoy.
This is a really useful mental model
to get your head around,
this idea of training the evaluator,
not just training the original prompt,
because now that we have an evaluator,
we can use this evaluator
as our evaluation metric for DSPy,
so we've gone from like a labeled data set
to a synthetic prompt evaluator,
which is really helpful.
It just expands the amount of tasks that you can do,
'cause it's usually easier to provide labeled examples
of an evaluator than it is
to provide labeled examples of a good joke.
That's much harder.
Yeah, feel free to use this.
Make a joke GPT if you want to.
That's what I used it for.
But yeah, this is really helpful.
You can also use this to generate more data, right?
So you could scrape a load of jokes on the internet
and then now use your evaluation metric,
and with GPT-3, you can actually tell like 85% accuracy,
which jokes on the internet are funny,
so which ones should you include in your dataset?
It's gonna really improve things.
All right, hopefully that's helpful for you
and opens a few doors or sets off a few light bulb moments.