-: Hey, I'm gonna walk you through DSPY,
which is a prompt optimization framework or library.
And it's pretty cool, actually.
Very complicated and a little bit over-engineered,
but I did get pretty good results with it.
Now, the reason why I wanted to cover DSPY is
that some engineers in the AI community have been saying
about how it is the end of prompt engineering.
And I, my personal opinion is that
even if you use AI to write the prompts, which is
what DSPY is doing, that is still prompt engineering
because you're actually engineering their entire
system around the prompt.
And that's the important thing.
Like how does the, what are the inputs,
what are the outputs, what are the evaluation metrics?
And DSPY doesn't automate that part.
That's what it needs actually in order
to do the automatic prompt rewriting,
I would say it is prompt engineering.
Prompt engineering isn't just writing the right combination
of words in the right order to get AI to do what you want.
It doesn't really matter to a prompt engineer, I think,
whether they wrote those words
or whether the AI wrote the prompt.
I'm just gonna walk through how to do this
and specifically piece this together from their
tutorials and documentation.
I'm gonna try and explain it a little bit better than
I could understand from the docs
when I looked at them initially.
I'm gonna go through how to use it in 8 steps.
So first of all, you wanna define your task.
I think generally, just before you take on the complexity
of a framework, just try and do it in the open AI API first.
So just gonna show you what I'm trying to do here.
I'm trying to get it to tell me a funny joke about a topic.
So this is just gpt-4-turbo using open AI API.
And you can see here "why don't fish do well on school tests
because they work below C level".
Ha, ha, ha, not particularly funny, but we're gonna try
and make it tell funnier jokes.
Once you've got the tasks to find, you want
to define the pipeline in DSPY
and there's a few different tactics you can use.
So typically you wanna start with the most powerful models.
I, again, I start with GPT-4 turbo here.
And there are different types of modules you can use
to do this type of analysis
or to get it to optimize your prompt for you.
One is like chain of thought, that's a pretty common one.
Tends to work with AI.
You ask the AI to come up with some rationale first
and then give you the result.
And then you could also use retrieval like rag,
a lot of their examples use rag,
although I didn't do that here,
where you could make a search
and give that context to the prompt when it's searching
or tool use as well.
You can also give the prompt, give the AI models
and tools to use and then
incorporate that into your pipeline.
That's all kind of complicated.
And their most basic example includes rag,
which I think makes it too hard for a lot of people to grok.
But I'm just gonna show you an example with a simple prompt
and some optimization.
First we're inputting DSPY.
The convention is you just keep it, you know, as DSPY
and then you can see wherever you're using it
in the script.
So first we're setting up GPT-4 Turbo
and this is just an open AI model.
So that's pretty easy.
And then just configuring DSPY to use that by default.
And then when we hit run,
you can see it comes back straight away
and it gives the same response.
It doesn't, it's not guaranteed to be the same response
as using the open AI API
because our temperature is set to one.
There is some randomness to the results,
but you'll generally find that
for some reason OpenAI always gives the same basic jokes
again and again, which really suck.
That's what we're trying to solve,
but at least we've replicated it here.
Alright, that's the first thing
to understand about DSPY is it needs
a signature in order to optimize.
And the signature is literally just what are the inputs
and outputs and you can define them in line,
which is pretty nice for small tasks.
And this is just prediction signature,
which is the most basic one where it literally just asks,
it gives it the inputs and and outputs.
And you can define it this way.
So I'm saying I have a topic
and I want to get back a joke and you can run that
and it works the same, which is pretty cool.
So now you have this joker thing is
basically like a little module.
You can, a little function you can run in order
to replace all this code here that we had for OpenAI.
Alright, so that's interesting.
Now you notice we didn't
actually have to write a prompt here.
This is the whole prompt and it actually writes the prompt
for you, which you'll see in a second.
The main convention that I recommend
and they recommend is to create a class based signature
because it's just a little bit clearer
and you can do more advanced things.
But this is the same example.
We create a class and we inherit from DSPY signature
and then we create a doc string.
This is the prompt, this is actually
all the prompt that we need.
Make a funny joke given a topic
and that's all it needs to optimize, which is pretty cool.
Then you have the topic of the joke is an input field
and we've specified that it's gonna expect
that it's gonna throw an error if we don't,
if it doesn't get a topic.
And then it has an output field,
which is the funny joke itself,
and it takes into account what's written in
these descriptions as well.
All right, so now we have the signature.
We wanna also create a module in order to be able
to run this signature.
The module is basically like, you can think of this
as a prompting strategy, whereas the signature is
what are the inputs and outputs?
And in this case, you know,
we're just using the basic predict prompting strategy,
which is, it doesn't have anything fancy,
it's just a normal prompt.
But you can, you know, once you kind using structure this,
you could also bring in any other kind
of prompting strategy you want.
So you could actually pull,
build together a bunch of different modules
and stream 'em together into a pipeline.
But ignoring that complexity, this is all you need,
you pass it, this is it here,
I'm creating one called COT Chain
of thought and we're just passing it.
The built-in chain of thought module here,
which is something that DSPY, so DSPY created just
to give you an outta the box chain of thought strategy.
But you can create your own.
And then, so it needs a self.prog,
that's how it makes progress.
That's what the prompt is.
And then you need a forward here if you think about it,
like in order to move forward I guess is the way you can
think about this, that you need a topic
and then you call self.prog
and then everything else is just inherited.
So don't worry about this if you don't understand it,
if you don't not into object oriented programming,
you don't really have to worry about it.
Just copy this example exactly, essentially
and then just change the signature
for the one that you're doing.
But this is what a chain of thought prompt looks like.
First it gets you to, it gets AI to create the rationale.
It's like how should it approach the result?
And then it actually asks to do the task, right?
But here it's, you can see it's getting
like a slightly better joke.
Why don't fish make good musicians?
'Cause you can tune a guitar but you can't tuna fish.
And so it's gotten better because it's got this rationale.
First, you're giving it more tokens to think.
And so this is a common prompting strategy, right?
Why did that run so fast?
By the way, one really nice thing
with DSPY is it locally caches everything.
If you are doing running this a bunch of times
and it does get expensive
because it does make hundreds of calls
to API when you optimize, I'll show you that in a sec,
but if you run it again,
it literally just looks at the cache first.
So you're never gonna cost yourself too much money
by running the same thing over and over again.
And it'll come back immediately if it's already
run, which is very nice.
All right, and the other cool thing is you can
expect the history of the model.
So here I've just got the last one result.
And you can see here that this is the actual
prompt that we sent in.
This is the prompt that created from our signature,
make a funny joke given a topic.
And then it says follow the following format.
It gives the topic of the joke.
Let's think step by step in order to, and then,
and then it inserts the specific thing it's trying to do.
And then it tells it what the variables, you know,
input and output variables are.
If I had one criticism of DSPY
it's that their actual prompts aren't very good,
but don't worry about that too much.
It does a pretty good job.
And you can adjust all this if you want to.
And you can see here recent, let's think by step
by step in order to, and then it,
then this green is like what it came back with.
Okay, hopefully that makes sense
to you if you're familiar with chain of thought.
But now we have a basic pipeline.
We wanna explore a few different examples
to understand the task a little bit,
and we have a topic of fishing
and here's a joke from Ricky Gervais, which you know,
this is an example of the type
of caliber we want to be able to train into it.
Give a man a fish,
he'll probably follow you home expecting more fish.
So that's a good example of a good joke for that topic.
But, and we can compare that
to our actual joke that it came back with.
And you can see that it doesn't do as good as
Ricky Gervais, that's to be expected.
All right, you wanna try other examples
you might run into errors with.
So I was a bit worried that, you know,
if we did a topic on drinking
or whatever, then it wouldn't, it would refuse.
But in this case it seems fine.
You could try other, you know,
more adversarial examples like trying to get it
to make a really taboo joke and see if it fails.
But this one seemed to be fine.
I didn't wanna make very taboo jokes, I just wanted
to make them not entirely politically correct,
but this is a joke.
It came back and it was actually wrong.
It says, "I told my friend I was going
to the bar for some fruit juice.
He looked confused when I came back with a beer.
I said, technically it's a bunch of grapes".
That's actually wine.
So pretty weird bizarre joke there.
And so we're gonna train it.
So far we haven't done any training
and this is where, you know, it starts
to get very interesting.
There are a lot of different strategies for getting data
for training and you can just do this manually, right?
If you just go and write a bunch
of jokes, you could do that.
But one really quick and dirty way to do this is I go
and find a bunch of resources.
So here I've just found a bunch of joke websites
and then I just copy and paste the websites into ChatGPT
and I ask it to give me an array
of those jokes in this structure.
And then specifically, I usually ask it
to just reverse engineer the input variables.
If you look on some of these websites,
you'll see it does have the joke, right?
And it does have the comedian Ricky Gervais,
but it doesn't have the topic on that website.
So I just ask it to write the topic, to guess the topic.
And then that way we have some training data now
'cause we have our inputs and then we have expected outputs.
But obviously we've just reverse engineered those inputs,
which is pretty cool.
And it does a really good job of doing this.
It's way faster than you doing it all manual yourself.
We can see I've got a bunch
of different jokes in this method.
It took about 10, 20 minutes to get these together
and that's all you need really.
You could also have it in A CSV and pull from a CSV,
but I just written some code here that goes
through all the different jokes
and then turns them into DSPY examples.
And it's important to have those examples in here
and to be able to actually in this format,
to actually train the data.
A few different things are important.
You could put the name of the comedian in here as well.
For now I've just said, okay, get me the topic and the joke
and then I pass them in to the example.
And then I say with inputs topic.
So this tells it that the input variable is topic
that you could also have, you know, multiple input variables
and then the rest of everything else it becomes like an
output variable or something else, right?
But this code basically splits it into three sets
and it took me a while to understand what they meant
by these different things,
but because they use slightly different
terminology than I'm used to.
But specifically the train set is
what you're gonna train the model on.
So that's your training data
and that should be the largest bucket.
The validation set, the val set,
that is when it's optimizing your prompt,
that is what it wants to test against
to see whether it's doing a good job.
And then the devset is left aside as like a final check
to see how well it really did.
So the reason why you split it into these three different
things is it needs something to optimize against.
So it needs to train on the data
and then see if it can predict something
and then change the prompt of prediction fails.
So that's why you need these two.
But if that's all you do, then it's seen all the data
that it's testing against.
I mean that's not good practice.
What's better practice is to see if it can generalize
to new problems it hasn't seen yet.
And then, and that's why you wanna train
and validate the algorithm with this, these sets.
But then do a kind of blind test on this new set,
this devset here,
and that's gonna tell you whether you've actually done
a good job or not, right?
You can see here we've got 76 jokes in the training set,
25 jokes to test them against and then 26 for the devset
and this just splits into, into 60, 20, 20.
Cool, you can use that, you can adjust those if you want.
But here's an example.
You can see there's the kind of formatted response here.
All right, so now we're gonna get into the fun stuff,
evaluation metrics and the,
you can actually do this programmatically if you have,
maybe you're optimizing a prompt for a blog post
and you can count the word length with a python function,
that's ideal because DSPY does a lot of testing
and therefore if it's expensive to run that test,
then it's a real problem.
the other ideal thing is if you have a human annotated
set of questions, if you're building a question
and answer bot and you get a human to go through
and label a hundred questions with the correct answer,
that can be really good as well
because you can use that as a test set
and that is something, again, it can check very quickly
and that can be really helpful in terms of the speed
and cost of running DSPY.
But in my case, it's coming up with the creative task,
it's pretty hard to measure.
Like I can't just,
'cause I said this joke is about fishing doesn't mean
that there is like a right answer
to telling a joke about fishing
and they might get a very different joke,
but it is actually funnier than the one that I annotated.
So in this case it makes sense.
Instead of using programmatic evals
or human evals, instead you should use synthetic evals
and that means using AI
to assess whether the joke is fun.
That is its own kind of broad topic,
which I won't go too much into,
but the nice thing about DSPY is it makes it simple.
So I create this again, just another DSPY signature,
elegant and just an assessment prompt
and you could optimize this prompt itself,
which is pretty fun.
But yeah, you wanna assess the quality
of a joke along the spec side dimension.
And then I have, you know, a few different questions here.
I have the joke, the topic,
and then I have the question to assess the joke against.
And then the output field, which is yes or no.
Now I have that signature
and then I just set up this function metric
where I just get in a topic and then I get the joke
and then the prediction, sorry,
the topic and the joke from, and the topic comes from gold,
which is the golden answer or the reference answer.
And then the prediction comes from the model itself.
So you need to set it up.
And then I have three questions
which checks the performance.
So the three questions are, is it funny,
would this joke actually be funny
to an adult attending a comedy show?
Is it relevant, is the joke relevant to the topic?
And then format is the joke is only the joke return,
no disclaimer or text pretending the joke.
Lemme just run this, and here I'm using
because I have, I set up this TJ pair example before.
And the inputs are basically just the inputs
that we decided before the topics and then the labels
and the label is the output, which is the joke.
You can use this structure here if it's formatted correctly.
But you can see that the TJ pair
that the first one we came up with
before, if we look up here,
it was the one about fishing.
So it's saying that passes the test, which is great.
It's scored one the, yeah, there we go.
So that scored one, it's a hundred percent.
And that means, the way I calculated this is I took
the score and then I divided it by the length of questions.
So it sums up like if it's scored,
it gets one point for each thing essentially.
And then I just round it down to two decimal points.
And you can see how it does this again, you know,
it's just inspecting history, it's assessing the quality
of the joke, it's giving the format.
And then you can see it came back
with the assessment answer.
Now one thing I noticed, I dunno if this is some issue
with DSPY in general,
but I found that the prompt format comes back wrong.
It's supposed to just say yes,
but instead it repeated the word assessment answer.
If you just follow the tutorials blindly,
it's gonna tell you just to check for the word yes.
But the problem is if you have this assessment answer
in the answer here, then you'll get a zero.
Even though it did answer yes.
So I had to change this code a little bit where you can see,
that's why I'm saying here if yes is in the answer,
rather than if the answer equals yes,
which is what they recommended in the tutorials.
So just a little gotcha to check.
'Cause it was saying my jokes weren't funny
and I'm like, I'm sure they are funny
and found out there was saying yes,
it was just not coming back at the right format.
So that's maybe something to look into, all right.
And then just to test it
and give a bad response just to check it.
Here's a topic, fishing,
and then I've, this is like one
of the typical things you get from ChatGPT.
It'll say, okay, here's a funny joke for you,
or it'll say there's a disclaimer, okay,
this is not very family friendly, but here's a joke.
It doesn't like to tell funny jokes
because it gets upset, I guess
because it's been trained to be very politically correct.
It, you know, it's afraid of offending people.
So it usually gives some disclaimer
or it tells you what it's gonna do
before it does it all right?
And that's a problem, so you can see here that we scored 33%
because we did have the disclaimer at the beginning.
And also the joke isn't very funny, but it was relevant.
So we scored one point, you know, out three, cool.
So now we've, this is a lot of setup as you can imagine,
but I promise you the setup is gonna be worth it.
So when you run this,
and this normally takes a bit of time, right?
Normally it takes a few minutes,
but what this is doing is evaluating
the actual devset, this is the,
what we're gonna test against.
It's establishing a bench line of basically like
how good are the jokes.
So it's using our evaluation metrics to see the jokes that,
how well does the normal prompt work basically.
And this is our make joke chain,
this is the chain of thought.
So this is without any optimization,
we haven't changed anything about the prompt,
how well does it work?
And you can see that it does 18 out of 26 possible points
or 70% right?
And you can see which jokes it failed on as well.
If you want you can opt, you can export this.
All right, that's established the baseline.
If we get better than 70%, we're happy, right?
But now after all this setup, we've now gotten to the point
where DSPY is gonna be useful for us
and we're gonna use an optimizer.
It might be a little bit confusing for you
to think about this, but essentially
because they have obscure names,
but essentially there's only really four optimizers
that are worth doing that there's,
and it really depends on how many
examples of the task you have.
So the first one is just called BootstrapFewShot.
And what that does is it, you can think about it
as just adding the right examples to the task.
So if they pass the metric,
then it adds it into the prompt as an example
of the task being done well.
There's also a BootstrapFewShot
with a random search and this, you know,
if you have more than 10 examples, if you have 50 examples,
then it's worth using because it actually does
do some optimization.
It will find the examples
that improve the performance the most against
your evaluation metric.
So it is more expensive 'cause it does a lot of testing,
but it does, you know, a better job than BootstrapFewShot.
And then if you have a lot of examples, you can use MIPRO.
And what this does is it does BootstrapFewShot
with random search, like it does optimize the examples,
but it also optimizes the instructions.
And it's important to note that BootstrapFewShot
and BootstrapFewShot with random search,
they do not change your instructions at all.
It's just uses what was in the doc string
and then adds a lot of examples of the task being done.
So that's the major difference here.
There's also one more we're gonna talk about
which just optimizes the instructions without
messing with the examples.
So we'll do that at the end, and then BootstrapFinetune
is really if you want to take a smart model
and distill it down to a smaller one if you wanna do some
kind of fine tuning essentially.
Alright, so let's see how this works.
And again, normally it takes a few minutes to do this,
but we end up with a compiled model at the end
and that compiled model
it supposedly will do a better job essentially.
And you can see because it's added some prompts as some
examples to the prompt, which we know is like one
of the prompt engineering principles of like how
to improve performance.
So you can see it already does a better job.
So it has the rationale here
and it says, let's play on the idea
of a novice fisherman misunderstanding common fishing terms.
And the joke is better, I think.
"Why don't fish make good musicians?
I took my buddy fishing and he threw his guitar in the lake
because I told him we're going to catch a few bass or bass".
So it's better, but like,
actually a little bit weird to set up.
Anyway, so I would say it's at least more
interesting than the first joke.
All right, and let's see what DSPY actually did for us.
All it did, again, it didn't change any of the instructions,
that's the instruction that we gave it.
All it did is it added these examples.
So it added this example and it added this example.
Okay, we can see the full thing added one on holidays.
And you know, it's important to note
that these synthetic examples, so these are things that like
these are things that like it is added.
So some of them are synthetic examples
and then some of them are actual,
some of them are actual examples
from your training data that's what it does.
So that's interesting and it does improve the performance.
So which we could test but I'm gonna show you
the more interesting one, which is the BootstrapFewShot
with random search.
And what this does is it makes sure it's including
only the best examples.
And so if we evaluate this, we can see
that the performance is a lot better.
We've got 94% instead of 70%.
So that's the one you really wanna use if you have more
than a hundred examples.
It actually really improved the performance
and we didn't have to do anything to optimize the prompt.
And we can see if we run this we can see which
this is the full prompt here, like which examples perform
the best when it added them in, right?
And some of them again are from the training data,
some of them are synthetic examples
and you can see as well adding some more offensive jokes
in here seems to have improved the performance.
But we don't have to know why one prompt
and one example performed better than another, DSPY
has figured that out for us, which is great.
Cool, and yeah, this is really good
and it's a big leap forward
and we didn't have to do any prompting.
Yes there's a lot of setup,
but once you understand how this works
and then you can just change a few variables
and then you've got it and you can save
this locally as well.
So you can use this prompt and you can load it.
So if you're using this in production,
you could just load the turbo.joke.json
and then you have the optimized prompt to use.
Cool, so hopefully that makes sense.
But we're gonna show you COPRO, which is just
to show you like we're not gonna do a full MIPRO one
where MIPRO is the one
where optimizes the examples and the prompt.
It's very intensive and wouldn't necessarily do it
if you are using a GPT turbo or expensive AI like I am.
'Cause it might cost a lot.
But we'll try COPRO and you can set limits on this as well,
but I just wanted to test it out and see how it works.
COPRO only optimizes the instructions.
So the way you set it up, it's very similar
except you pass it the eval directly into the prompt
optimizer to compile it.
If you look here, oh here we go.
So it is actually changing it and making new responses
and you can see that wherever it has tried
that prompt in the past it did just jump ahead
and then ones where it hasn't tried that prompt in the past
and then it goes and runs it again.
So this is again, one of the major benefits
of using DSPY is that it saves you time
for retesting something that's already been tested.
But you can see here that it's, you know,
live updating the score.
This one, it's got 74% response, which is pretty good,
but you know, it's doing okay.
This is just changing the instructions, right?
It's not adding any examples to the prompt,
but you would get a better result if you changed both
the instructions and the prompts.
But it takes a long time.
And you can see this is the previous winner that we had.
We just got that through the candidate programs.
You can see that the prompts
that it comes up with are pretty weird.
So it says like engineer humorous remark related
to your specified topic with strategic use
of satire understanding hyperbole
for a heightened comedy effect.
It just comes up with really odd prompts,
but it does, it tends to work.
And this one got 86% with just improving the instructions,
no examples added.
So you can see why it would really improve performance.
Now a couple of things to be wary of with DSPY.
It does make hundreds of tests,
so it can cost you a lot if you're using AI
to test whether it did well or not.
So be careful both if you run it,
especially if you're using GPT-4 to evaluate
and it might be cheaper just to optimize it yourself.
But yeah, hopefully you understand how this works
and you can use it for optimizing some of your own prompts.