Dspi is an unusual framework, right?
Uh, actually met up with the founder last week.
Like, really smart guy that came out of Stanford.
And they've used it internally at Stanford for a few of the other projects like storm, which was like
precursor to deep research, was built on DSP.
Oh, interesting.
So I'd say DSP is like one of those kind of open secrets that a lot of people are using.
Like someone went sleuth and found that all of the major vibe coding apps replit windsurf, like all
of the CEOs, follow DSP, but they never talk about DSP.
The reason I think is that the documentation is very academic.
It doesn't really talk that much about real world business use cases, and it assumes that things like
what, like Multihop is, or that you're familiar with this specific academic data set that nobody uses
in the real world.
So once you trudge through all that, you'll find it's actually pretty simple.
Okay.
They just looks very complicated because the people who made it are really smart and therefore the they
have a bit of a blind spot, I think, in terms of making it more accessible.
Yeah.
They also settled on this weird domain specific language that is not very pythonic.
Here is a Dspi signature, which is just if you extract it out, it's literally just defining what the
inputs and outputs are of your prompt.
Okay.
And and these are the instructions right.
Extract structured information from text.
Right.
So you can give really simple instructions.
And that ends up going into the and that ends up going into the prompt.
That's weird.
Yeah really weird.
And then you have to specify the types.
And this is like despite input fields Pi output field.
Right.
Once you've created a signature that is your prompt and the it builds the prompt from that.
But once you've got it in that format, you can swap out models really easily because you just it's
literally the same signature and it has this adapter underneath that kind of makes it run on any of
the different llms.
So you don't have to worry about how does Google do formatting of inputs and outputs?
How does OpenAI, which is nice, but the really big thing is the optimizer.
So once you've told it, here are the inputs, here the outputs, and you've given it an evaluation
metric and a data set to evaluate on it can automatically improve the prompt.
So that's the real payoff.
Yeah.
Once you see that magic in action.
Okay.
Now it's worth it to me to go through all this extra effort.
Yeah.
And you can regularly see it pretty easy.
Like a ten point percentage increase.
Like you can go from, say, 40% accuracy to 50% accuracy.
Or in some cases, I've gone from 50% accuracy to 90% accuracy.
The really difficult thing about DSP as well is just that it forces you to really think about what you
actually want out of your program.
And quite often when people struggle with DSP, it's actually because they're struggling to specify
what they want their program to do.
What is the evaluation metric I want to use to define whether this program is working or not, and formalizing
an evaluation metric.
Most people, when they're building AI tools, are doing vibe based right?
This is what I do and this is what you should do.
Like in the beginning, you should say, hey, I think the output looks good or I think the output looks
bad, or my CEO thinks output looks good or he's noticed in these scenarios it does badly.
So you start with vibe based and it's very hard to go from like my CEO likes this to we have a formal
data set of inputs and expected outputs, and we have an evaluation metric that can check whether the
outputs are good given the input.
So like that is actually the hard part.
But that's the hard part of building AI applications.
It's not his fault.
Yeah that makes sense.
That makes sense.
Yeah.
That is like where my mind trips up.
I'm like, okay, so this is describe something.
But I'm used to like being in the details.
And this is actually saying, no, don't be in the details.
Let us do the details.
But I'm like but like, how do I even steer it?
I think a lot of the tension comes from the fact that they've almost accidentally built the world's
greatest prompt optimizer.
Yeah, but what they really want to be is like a full framework.
They want to use this domain specific language and not care about the details, because you shouldn't
worry.
You can just compile the program with a different model if you want, and don't worry about what the
program did.
Like just worry about the whether the outputs have good evals or not.
Yeah.
So one question because length chain is also oh don't worry about the details.
But like at some point no one was even understanding what was going on.
And no one was like, yeah, it does something, but no one knows how it works.
I'm a little bit like worried that it's like a length chain thing, whereas yeah, sure, like we promise
a great library.
But if no one knows what the hell is going on, what's the value?
So I'm curious what your take on that is as well.
Exactly.
Yeah.
So I use language chain a lot early on and then I like started digging into it and I was like, wait,
hold on.
They've just wrapped the length, the Python length statement like in the 100 lines of code for no reason.
Yeah, I always have this love hate relationship with.
But I would say where DSP is fundamentally different from long chain is that long chain is like just
use our framework and it will make everything easy.
Don't worry about the details, but sometimes you really do have to worry about the details.
Yeah, but they're not really giving you anything in return.
DSP is giving you the optimizers.
You genuinely can ignore the details if you have a good evaluation metric.
And that's the key.
And I would say where DSP is really useful is if you have a formal evaluation metric that you trust,
and if we have one which is dated, guess the category correctly, which is a categorization which is
super easy.
So I would say probably for a use case like that, that's probably ideal, right?
Exactly.
Yeah.
There are a million different ways to build AI applications.
And what I found is because I was working as a prompt engineer for the past few years, is basically
like a single pattern that you can just use and ignore everything else.
This is the evaluator Optimizer pattern, and what this solves is quite often the task you want to do
is fuzzy.
Like, there is no like, real formal evaluation.
If you're doing a blog post generator you could generate, you could check the length of the blog post.
But like, how do you check whether it's a good what is the what are you optimizing for exactly?
So the way I would approach that typically with clients is I would say, okay, who's your domain expert?
Or it might be the CEO, it might be someone else.
So like I worked with a team of psychologists, for example, to do like a personality quiz type thing
and at that time is really valuable.
So you don't want them like manually evaluating every single response.
What I started to do was this LLM as a judge type pattern.
Don't worry about eval ING the actual generator or the optimizer in this case, like the thing doing
the task.
Instead, build a judge to replace the domain expert.
Yeah.
And so you also normally or is it like prompt.
So you prompt that or you fine tune that or you actually use the spy to.
Exactly.
So that's also a DSP program.
So you just.
Yeah.
You do that as well.
Yeah.
Yeah, exactly.
So like the judge will check whether it passes the test.
Yeah.
And it'll give you some sort of score.
So with my DSP metric like the LM judge is my DSP evaluator.
So it allows me to do the fuzzy as long as the judge agrees with me or agrees with the domain expert.
Most of the time, 80%, 90%, then you can trust it to do the optimization and it doesn't cost too
much, especially if you can get the judge working with GPT mini or one of the cheaper models.
And it's usually possible it's actually a much easier job to evaluate whether something is good than
to create something that is good, like P and p equals NP.
And this is in a sense of where you need LLM as a judge, because obviously categorization you don't
need that.
Yeah, exactly.
So categorization is like a judge type task.
The like the fuzzier type tasks, like the example I'm using today, I've got like a notebook that I
could share afterwards is a telling a joke.
Getting an AI to tell a funny joke.
If you ask it, it just uses dad jokes and funny, but like, not real.
Like stand up comedian type jokes.
The first thing you want to do is train a judge to check whether the joke was good or not.
Like just 1 or 0 is really simple.
It could be really simple.
Are you always binary or is it just truthful?
I actually found binary works much better, and it's actually better to just stitch together a bunch
of binary evals than it is to use a Likert scale, or rate this one out of five, because llms they
tend to be too positive and it's like everything's a four.
Yeah.
So yeah, I just do one and zero quite often and you can build up more weights and you can say, I don't
know if you're generating like an article for every, you could say, what are all the different things
that I care about in terms of a good article?
Like it needs to have a catchy hook at the beginning.
It needs to be this length.
It needs to be.
You can actually string all these things together into one master eval and weight them and say, okay,
like catchy hook is 80% of the value, whereas length is like 10% or whatever.
So that's the way I think about it.
Cool.
So let's jump into the code, install DSP.
And then what about sentiment like positive negative neutral or is that too much.
It should be.
The way I see that is that is still like a in a way like a classification.
Okay.
So okay.
Okay.
Yeah I would say it's important that it's mutually exclusive and collectively exhaustive.
As in there's no ambiguity between.
Yeah.
Exactly.
Yeah.
Okay okay.
Makes sense.
Yeah.
The way you set up DSP is very easy.
It's just provider forward slash, GPT or mini or whatever one you're using.
You can set up multiple and you can use one as the teacher for the optimizer.
And one is the student to do the task.
I'll show you that in a second.
But you just generate it and you can put your prompt in.
So you actually can just use DSP as like a scripting language, right?
You don't have to worry about any.
You don't actually don't really need to use it like for optimizers.
Right.
And so I use it quite often now for like throwaway programs and like Python experiments and stuff,
which is nice.
There's a couple of nice features that DSP has straight out of the box.
One is that it's probably the easiest way to get something working on Azure or AWS or something.
It's literally just add a couple of more environment variables.
And and also, if you wanted to see how this run on another LM, you could just add in anthropic slash
chords on it.
Right.
It also has caching built in.
You can see how fast that ran.
Right.
But if I maybe change the temperature.
So it has like default zero temperature.
If I want one it'll run again.
Right.
Because I've changed one of the parameters.
So it will skip the cache.
And you can also do cache equals false if you want.
But but you see how fast that just loaded that from the cache.
Right.
It can save you a lot of money when you're doing experiments.
Even if you didn't remember that you had used that combination before.
Yeah.
Or optimization.
It's it happens to have a cache hit.
It will it will just use the cached version like it won't cost you anything.
Cool.
So that's interesting.
Then to create a program, you can actually just do it in one line if you want to.
So this is a basic joke program.
It just takes a topic and then gives you a joke, right?
Everything on the left side of the arrow, it just turns into an input variable and everything on the
outside it turns into an arrow.
Is this a DSL like input?
Exactly.
They've just added that as well.
Again, extra confusing, but for convenience is quite fun.
And if you wanted to say, I don't know, a comedian, you could do that.
And then literally like the program would just work.
So I really don't like that.
Yeah.
It's terrible.
So I will use I use Ruby, I don't like Python, so I will use dsp y RB which is oh yeah, it's in Ruby
which has a way nicer way to do this.
They don't have a DSL in strings, they just have it.
Yeah.
Ruby I think so.
Yeah a couple of hacks though.
Workarounds for doing it the more pythonic way.
You can see this is the type of dad joke it has.
Why do Python programmers prefer dark mode?
Because light attracts bugs.
It's a classic now.
One of the fun hacks to get use out of DSP is you can just see what prompt was run the last.
So this is n equals one.
This is the last thing that was run on the LM.
And that's a global thing.
Um and so you can see the actual prompt here.
This is what goes in.
Actually it's probably better if you do the scroll down.
It's turned that whole thing into literally just the system message.
Here.
Your input fields are topic.
Your output fields are joke.
And it's given the types all interactions to be structured in the following way.
By default it uses like this kind of markdown type thing, um, rather than like JSON, but you can
change it to JSON if you want.
This does tend to work better actually.
It's like JSON mode.
I think it ends up making it a little bit less intelligent as what people have found with testing.
But yeah, then put in the user message for you and then you get the response.
The really nice thing is it's all automatically formatted and then all automatically passed afterwards
as well for you.
Do you look at these like normally before running or like to see if you can optimize it?
And how do you see if it's good or not?
I used to really hate this because I don't like the style that they use, but it does.
Now I just go, I just use whatever they have like straight away and then it tends to work okay.
So yeah, I manage my like really important prompts like the core prompt rally is not using DSP, but
like I run all of my experiments with DSP and then I find something that works.
And then I'll incorporate that into my main prompt.
Yeah.
So because it's in this format it can go and optimize this.
So one of the optimizers adds few shot examples.
It will test a bunch of your examples that you give it.
And you'll see which ones are the best.
And it'll add them and but it automatically puts it in as user message response user message response.
So you don't have to do all these convenience methods that you might have to do normally, which is
quite nice.
Cool.
Um, so the optimizer you run first and then you store that version or something like that.
You can.
Yeah.
Exactly.
Yeah.
But this is my way of, like, more pythonic way of creating DSP programs.
I just create the fields and the instructions.
And then there's this, like, very deep in their library, there's like this DSP signature, which,
like, doesn't appear in the documentation anywhere, but it works.
And so you can just give the signature name of whatever you want, the instructions and then the fields.
And it does use all of this.
So like actually the like some of the optimizers are program aware as in like they it shows itself what
the program looks like.
So even the name of the program and stuff.
So like what you name things is actually super important to the DSP.
It's funny because that's how the Ruby version works.
You have a class that is the signature with an input and an output and a description, and you run that,
which makes more sense to me.
It's like this.
Yeah.
So you see here I put Dot predict.
This is just like the base model train of thought or whatever there as well.
But you can change that to change the thought.
And then now you have.
Yeah.
So I've just printed output joke.
But if I just print what is output.
It's a prediction object with reasoning and joke.
So it's the reasoning that's the really cool thing is like they have a bunch of they have the react
pattern for agents to use.
They have sampling where it just generates five versions and chooses the best.
You know, they have things like that.
So they have a lot of these prompt engineering techniques that are built in, which is quite nice.
And it's just a one liner to add them, which is useful.
Cool.
Ever use multiple predictors and then go from there like in a multi or.
Yeah exactly.
Yeah.
So you can basically once you've created the signature like then any valid Python object can be a module.
Essentially you can string loads of different predictors together.
Create like complex workflows.
That's where you get more into the long chain type stuff where yeah, like multi-step and like synthesis
and all that stuff.
Yeah, exactly.
Yeah.
Just to show you, like I just brought in Gemini Flash and all I needed was my Gemini API key.
I don't need to worry about how Google's currently doing stuff, which is batshit.
So all good.
Really nice.
And you can see here, by the way, you can have a globally configured like I configured the LLM up
here somewhere.
Yeah DSP configure and it would just use that one automatically.
But you can also just run it in context.
So you say with DSP context LLM equals Gemini LLM.
Then it will just use Gemini for anything in there.
But like the OpenAI still config.
Yeah.
And some of the modules and stuff that you can pass in the LLM.
But I found it's inconsistent actually.
Yeah.
So yeah you could do different steps in pipelines with different models and stuff.
Exactly.
Yeah.
This is like a chain of thought.
Example, which we've already put in there.
And you can print out the reasoning.
And because it's all automatically passed, you can just do like output dot reasoning output dot joke.
Yeah it's quite nice.
So you really don't have to worry about that.
And is there does it is there a cycle feedback loop where you give feedback and it looks at the reasoning
and sees if it can optimise from the feedback?
And the reason.
Yeah.
So some of the optimizers work that way.
Yeah yeah.
And I'll cover that.
But it is it's more of a static thing.
Like you run an optimisation job like there's no active learning.
Yeah.
But for example if I have like hundreds of people saying things, this summary was like needed something
more like that or something like that.
And I have the chain of thought and everything also saved.
I could run it once a while to update the prompt with an optimizer to like, optimize for that output
and more.
Yeah, exactly.
Yeah, yeah.
And, and that you start to get into this really healthy, kind of virtuous loop where because you have
the eval metric, you can either make the eval metric perform better or to cover that, that new use
case that you've seen coming up.
And then you can run the optimization job again.
But you could also go back and change the way the program works.
You can start from yeah, you can use a different optimizer altogether.
See if that works or.
Yeah, or try it with Gemini.
Gemini does better or whatever it is.
Yeah.
Cool.
The again, this is like stupid.
But the way it works is this is actually how you build a real kind of DSP program.
You you create the class, you know, with the this is the instructions.
And then this is the input field.
This is the output field.
And then you create another class for the module.
And it took me a long time actually to kind of get my head around why this is necessary.
But essentially this allows you to build programs of arbitrary complexity.
The DSP module is the base class that chain of thought and predict inherit from.
So it's a way of you to add your own prompt engineering techniques to it.
So you just define an init and it inherits from from the module.
And then like in this case I just create the joke generator.
Right.
And then you just have to define a forward method.
And the forward method basically just takes the inputs in this case just topic and then gives you the
outputs you can pass.
So this is a simple one right.
It just does the prediction and then ignores the chain of thought and just returns the joke.
Yeah.
But you could call another predictor or do whatever you want there as well right.
Exactly.
You can see that the, the yeah.
Like the I think the result is basically.
Yeah the same.
Why did Python script need therapy.
Because I had too many deeply nested if statements can handle the indentation.
Yeah.
But yeah the so that that's like the just getting a script working.
This is where it gets really powerful.
So here what I'm doing is I'm training my optimizer because I'm going I think, I think this in terms
of we used to have test driven development.
Now it's eval driven development.
Like you have to formalize your eval.
And if you formalize your eval, then you can train an arbitrary program, whether it's fine tuning.
And actually GSP does fine tuning for you as well.
They have this like Better Together optimizer, which does both fine tuning and optimization.
That's crazy.
But like you actually, funnily enough, like I find that prompt optimization almost always beats fine
tuning.
Yeah, I've never had fine tuned work in my work.
Yeah, unless you've got like thousands like I think the there was a paper that said that until you
have 2000 data like inputs and outputs, lines prompting beats fine tuning.
Okay, great.
Yeah.
What I did and this is another little hack that I do to get to build a data set, is I just went online
and found a lot of funny jokes and just I used a deep research here and then told it to give it into
Python.
So I got like the topic, the joke, and then the comedian in here just for attribution.
But so I got, you know, I got it to create.
I think it's like you look like Ricky Gervais if you.
Yeah, I'll tell that once, actually.
Yeah, I'll take that as a compliment.
Not that you're saying.
Oh, he's great.
And then.
And then I took the unfunny jokes quite often.
The task.
It sounds like a complicated task to build a data set, but quite often it's just that you need to go
and pick out, like, select things that you think are good.
And you didn't even have to really explain why you think they're good.
And then you just have to.
Then as the things that are not good, you just have to generate a bunch of GPT answers because that's
what you're not good from.
You're trying to get it to not sound like ChatGPT.
Yeah.
So here I just I literally just got, I said to ChatGPT mini just to make sure it was especially dumb,
like, just give me a bunch of jokes on these topics.
And, and then so that's how I got these.
So that's a really good hack for building a arrested.
It held up a pair of pants.
So.
There's a couple of useful things to think about here.
And the name is inconvenient.
Like in the documentation.
They don't really explain like what it means or why you would have a training set versus validation
set.
But as as best as I understand, training set is the data that you give to the optimizer to train on.
Like it can see that data.
It will evaluate that data.
And it will use some of those examples when it's optimizing like it'll add them as few shot examples.
Right.
Validation set is you give that to optimizer as well or you don't have to, but you can give it to optimizer
as well.
And it uses that as the test set.
So it will try a bunch of it'll look at the data, try a bunch of things and then it'll run a test on
the validation set.
If you don't give it the validation set, it will just run it on the full training data so it can end
up costing quite a lot.
That's one of the primary reasons.
Also, I think the results don't generalize as much if you don't have a separate test set.
And then the reason why you need a development set is that because it also sees like the test set,
when it's running the results, sometimes it will bleed across and it won't give you like really good,
like generalization, like you'll get good optimization scores.
But when you try and run a new joke or whatever, it won't be very good.
Um, and then so the development set is what you use.
The optimizer never sees.
Right.
Like you just create a new like you just run an evaluation yourself after the optimizer is done.
And that's, that's really important, I think, for proving to the client or proving to stakeholders
that like, it actually did do a good job.
It's not part of the loop you need.
Yeah, you need to have the evaluation step in in the end.
But it's just running it.
It's not training or optimizing or doing anything.
Exactly.
So the way I split it usually is, I'll try and get like more than 100 examples or 200 examples of good
versus bad, and then I'll split it 60, 20, 20.
Yeah.
You do new examples I assume, right.
Yeah.
Exactly.
Yeah.
So you can see here we split this out a couple of weird things.
The way that GSP wants the examples is also like a bit odd.
So you see here it's despite example.
And then you just give the different parameters in this case because it's the judge we want.
This funny equals true to be the output.
And then you just do dot with inputs and then list the ones that you want as inputs.
And then it will assume the funny is the output.
And examples if you always use examples for optimization right.
Or yeah that is what like a data set is for.
That is the whole point right.
That's how they format it.
Yeah exactly.
Yeah.
And they have like internal stuff that will turn that into a pandas data frame essentially.
Yeah.
Cool.
So we've got our data set.
Just here's an example.
We've got the topic whether it's funny or not.
So we create our judge.
And this is a case where like actually it's convenient to do the inline thing we're just taking because
we don't care that much about this judge.
We just care about whether it's accurate.
We don't care how it gets to accuracy necessarily.
So we're just giving a topic a joke.
And then we're saying whether it's funny or not.
You can also set types in here as well.
So I set this as a boolean because I found that otherwise it gives you back like a big spiel of like
why it's funny or not funny.
You can see here like in this case judge says it's says it's funny.
True.
Actual ground truth is false.
So the judge was wrong here and I made it a chain of thought.
Because chain of thought tends to really improve the results of judges.
It thinks it's funny, but it's not.
So we need to train our judge so we can actually do an evaluation of our judge.
Evaluate.
Convenience method is like really helpful because it runs in threads so it runs in parallel.
It makes it run a lot faster.
So it'll run eight of them at the time in this case.
But you can set that to whatever you want, which is nice.
And then you just pass it in a metric development set as in what the.
This is like the one that we kept aside that the optimizer won't see if that makes sense.
The final judgement.
And we can see that here.
It's 51% accurate.
And you can see the actual examples here.
This one wasn't funny but the judge predicted.
True that it was funny.
So it got that wrong right.
This one was true.
It was funny.
And it predicted that it was funny.
So it got that one correct, right?
The nice thing about judges is that they can have an exact match score.
And the way I've set the prediction metric up is again specific to DSP.
You have to have this like trace equals none.
So you can set that.
So that allows you to basically set like a different metric response.
If the optimizer is evaluating versus if it is like learning from the failures.
So if the if trace equals is not none then then it's like learning.
So you could give it like a more strict evaluation criteria.
Because the reason why you would do that is like in this case it's just exact match.
So it's easy.
But the reason why you might want to do that is maybe if you did have an evaluation metric, which is
like rate this one out of five, you have a score then.
But when the judge is learning, it's deciding what few shot examples to put in.
So you might want to say I only want five star ratings as my examples, right?
So you don't have to do that.
But that's why that's there.
And it automatically will pass in the prediction.
And then the gold, which is like the essentially the the ground truth like from the data set.
So in this case we're just checking whether the prediction funny equals gold funny.
So that's like the simplest possible metric.
Cool.
So we have a judge that's wrong half the time.
How do we make it better.
This is where we're bringing in optimizer.
And this is the simplest optimizer.
It's just called bootstrap.
Few shot.
All it does is it just adds in a few or actually creates a few shot a couple of few shot examples and
then checks them against the evaluation metric.
If the evaluation metric is correct, then it will, then it will pass.
It will add them in to the prompt.
So when you run that you think it synthesizes jokes to She pauses.
Yeah, exactly.
Yeah.
So yeah, here we go.
So we have Max bootstrapped demos and Max labeled demos.
Max label demos is like demonstrations, as in examples from your data set.
So in it's allowing you're allowing it to add up to 16 few shot examples from your data set, from your
training data set.
And then Max bootstrap demos is you're allowing it to generate new examples.
Yeah okay.
Data set but still pass your metrics.
So would you ever run this without examples.
Or you will always add examples because you always add examples.
Yeah.
Because it needs I guess you could just make the max label demos equals zero.
And then it should be able to do.
But yeah I mean it's but you would never do that.
That's just not a thing.
Yeah.
Like I would even go to the length of I would go and run the program like 100 times to generate the
data set and then run.
Yeah.
And then you manually pick or whatever examples.
You can also set a few things like so you put the metric in there.
But you can also set a threshold so you can say I just care about 80% accuracy.
So if it gets 80% accuracy, stop optimizing.
You could also set the teacher so you can say I want GPT.
I want O3 to be the teacher and I want but I still want mini to be the task.
And so it's distillation.
Whenever it generates a bootstrapped demo, it will use GPT three to generate that.
Oh sorry.
We use O3 to generate that.
Yeah.
So that is like quite an effective approach.
And how do you choose the number of examples and what examples.
And when do you change the examples?
I find it's a it's like a very much an art rather than the science.
What do you do.
I found it like really weird results sometimes.
But yeah, broadly speaking I'll try it with the default and then I'll diagnose.
So there's a couple of different optimizers, by the way.
This doesn't change the instructions.
This still keeps your instructions, but it only changes the few shot.
And usually that's enough actually for classification tasks.
I'd say typically I would say that you want like at least five, maybe more like ten examples in the
final prompt, maybe more than that in order to get good accuracy on a classification task.
But you don't want too much.
Obviously there's a cost to running this.
I just this just happened instantly because because I have this is a deterministic optimizer.
And and because I have already run it before it pulled from the cache.
But like this can actually take ten minutes to run or something.
And it can cost like it can make a few hundred API calls or a few thousand API calls.
So it can actually cost you into the hundreds or maybe even thousands of dollars, depending on how
big your data set is.
Yeah.
So with Dspi, it's forced me to just basically be as dumb as possible, don't care about the prompt,
like just do the inputs and outputs and see if it works straight off the bat.
If it doesn't, then run an optimizer, see if that works.
If it doesn't, run a more powerful optimizer and then go, okay, I'm going to try and run a few experiments.
I'll do it with 16 demos, or I'll do it with five demos and just see what works.
So I only apply the complexity like on demand.
It's almost like just in time edition of complexity.
Yeah.
And like, for example, you made one last year or like half a year ago.
Do you ever go back to the example because people do give feedback and things do change and models change,
like when do you decide to redo this or add other examples?
Yeah, I would say typically once you like, I would rerun a prompt when when we discover a new dangerous
edge case.
Oh everyone's complaining about this now.
Yeah.
So then I would do a job just specifically to solve it.
Or if I've collected more than 50 new user responses, because you could have a whole system here where
you have a thumbs up, thumbs down button on your on your.
That's what I have.
So I'm thinking and then you that generates the data set for you.
And then you just rerun and see how well it does.
Yeah.
But yeah like it is like a continuous thing.
Yeah.
You could.
The dream is what I want to get to with our system is just running overnight every night.
It just keeps improving.
Yeah, and I don't have to worry about it.
Yeah.
And really, it's like the example.
So how but how do you then choose the examples or like you just give all the examples and say you figure
out this optimizer chooses the examples for you.
So you don't really need to deal with any of it.
You just say whatever the default is 10 or 16, whatever that is.
Exactly.
And I find for simple tasks bootstrap, few shot works pretty well and it's the cheapest one to run,
so it's like very easy.
It's pennies you bootstrap few shot random search works much better for more important tasks.
And again it only chooses examples, but it adds like a random search component.
So it will go and you'll find a new like like this one just goes through the examples and chooses one.
But it might it be at a local maximum, like random search actually just literally hops around your
data set and like picks out.
So it's more compute intensive.
But it does a really good job.
Yeah.
And then so that's optimizing the judge.
And here we go.
Obviously the eval would run a lot slower if it wasn't cached.
But we have even just adding a few short examples like we have 92% accuracy now.
Yeah that's cool.
Wow.
So that's amazing right.
Like you just go.
We didn't even have to think about it.
We didn't even look at the prompt.
And now we've had an 80% improvement with a couple of lines of code.
That's the magic.
Yeah.
I'm going to for sure use this for categorization like next week.
Yeah.
So categorization is perfect.
But then every fuzzy problem also is a categorization problem, right.
So now like the judge is the categorization problem, we can now use that to make our and you'll see
it won't work as well.
And it's more intense.
But it does lead you to a path to get the fuzzy tasks under control.
So now I've just created a new metric, which is the judge score.
And again, literally just saying the judge like I'm using my bootstrap to optimize judge.
Right.
Like this program that we created.
And that's the optimized.
That's the optimized.
Yeah.
Exactly.
Right.
That's the final optimized judge.
Right.
Yeah.
That we created here.
And then I'm just using that as a program.
Right.
So I just get my judge result.
So I pass in the topic and the joke.
And then I give a 1 or 0 right.
So that's really great.
And then you can create a data set.
In this case I basically filtered out the examples and I changed the inputs a little bit because before
if you remember we had the topic and the joke.
And then we check whether it's funny or not.
And in this case I'm just passing in the topic and then getting a joke.
Right.
So I had to create a new data set, the topic data set, and the only input is topic.
And then it would generate a joke.
The other thing I did is I filtered it.
So I'm only giving few shot examples where the joke was funny, because I find that it's actually really
detrimental sometimes to give negative examples.
You want it to only have positive examples, because when you give it negative examples, it tends to
follow that negative example weirdly.
It's like to say, don't put your hand in the toaster and it goes and puts its hand in the toaster.
Yeah.
But you can experiment on that.
Okay.
It's like edge cases.
How would you choose to include edge cases in the example or like you just include them and it will
be handled.
Yeah.
What I would do is I'll take the edge case.
I'll get your domain expert to rewrite what should have been the correct answer, and then use that
as the flu shot.
Yeah, because otherwise it's confusing or like it will throw it.
Exactly.
Yeah.
Or you could do like a rewriting type prompt, which would be again, another DSP program where it takes
the wrong answer and rewrites it to be the correct answer.
And then and then you can see in terms of optimization that.
So what I've done here is this is taking the topic dev set, which is those those examples that we had
before.
So literally just the dev set and it's just given the topic.
So it's just taking those topics and has found that out of 51 topics, only 2020 of them resulted in
funny jokes.
So we got 39% funny score from our joke generator, a baseline, right?
So now we're using the Big Guns Me Pro, which is like a Bayesian optimization algorithm that very smart
people worked on.
And this not just does the optimization of few shot, but it also changes the prompt instructions.
You can actually just run this as a prompt instruction optimizer if you want, by just setting max bootstrap
demos to zero and max label demos to zero.
In this case, I set it like you can also do light versus medium versus heavy.
Like I said it heavy because it's non-deterministic.
You want to set a seed as well so that like you get the same results and otherwise every time you run
it, it's not going to be any worse or better.
Yeah.
You can also set the initial temperature so it starts at that temperature.
And then it will try different temperatures at other sides and things like that.
So it also optimizes those parameters for you, which is quite nice, but pass that in and then I must
have messed something up because it's running again.
There we go.
Great.
Yeah.
Yeah, exactly.
No, sorry.
It was just running through the cache.
Cool.
Yeah.
No, I mean, I'm using mini, so it's fine, but that could be like $100 mistake if you're using O3
or Pro or something.
Yeah.
Um, yeah.
Just to show you, like, again, a lot of this won't make sense, but you can see it's super optimized
instruction texts like the few shot examples and then balance between different optimization strategies.
First it starts like bootstrapping traces meaning it's like it's generating new few shot examples,
and it'll have a certain number of attempts to see to get like a funny joke.
And then it's like doing that a bunch of different rounds and then it's proposing instruction candidates.
So we use the few shot examples from the previous step.
Generated data set summary a summary of the program code.
So it's like program aware.
It can see what the code looks like of the full module.
And then randomly selecting prompting tip to propose instruction.
So like you can actually override those as well if you want to subclass Dnipro.
But I'm too afraid to try and then.
Yeah.
So so it's proposed instructions like it did have a funny joke about the topic, but then it came up
with this one.
Generate a humorous joke related to the specified topic, suitable for general adult audience.
Be mindful of potentially sensitive content.
Right.
So that's its prompt its new prompt.
And then which model is it using to generate this prompt.
So by default it's just using the same model that you passed.
But you can set that.
So you can choose the prompt model.
You can choose the teacher model.
Right.
So the difference is the prompt model writes rewrites the prompt and creates candidates.
The teacher model generates new few shot examples.
Okay.
Yeah.
So it goes through all this.
And you can see it runs for a long time right.
Lots of trials here.
You can set the number of trials and stuff.
But once you have that you can run the optimization and then.
Yeah.
So that's actually better for some reason.
Maybe, maybe I didn't run it with that seed before, but yeah, it actually went up from 39% to 49%
here.
It got it did get a lot funnier.
It still does a few dad jokes and stuff like that is what I found.
But the way you would go to improve this is you would improve the optimizer, right?
Because we only had 90% accuracy.
But and we also had pretty easy examples where the dad jokes were very obviously bad.
And like the comedian jokes are very obviously good.
So we don't really have that many in the middle.
So I would go a bit more fine grained.
Now if I wanted to improve it further, and I would give it some harder examples to train on the jokes
that you think, yeah, yeah, yeah, I'll show you that in a second.
But this looks great.
Show me.
Yeah, show me the money.
Yeah.
This is like the basic one.
So the topic was Python.
Why do Python programmers prefer dark mode?
Because light attracts bugs.
Me is like, why did the programmer quit his job?
Because he didn't get a raise.
It's not funny.
But that's better.
That's better.
Yeah.
So this one's a little bit better.
Why did the coffee go to the police?
They got mugged.
I like my coffee.
How?
I like myself dark, bitter and too hot for you.
Okay, this is a bit better.
Why did the bicycle fall over?
Because he was too tired.
And then he said.
I hate when I lose my motivation to exercise.
It's like, where do these extra £10 keep coming from?
I think that's much better.
But yeah, anyway, so you can see that it is a little bit harder to get like for these fuzzier tasks.
But we did.
It's not to be sniffed at.
Right.
We went from 39% funny to like over 50% funny, almost 50% funny.
So that obviously you keep working on this, but like the fuzzier tasks are just harder.
Yeah.
No, I mean that it was definitely interesting.
So the way what I'm taking from this is the judge is like everything.
Yeah.
Actually my I'm like radicalized on this to the point where I actually don't think data matters.
Everyone says it's all about your data, your unique data.
I'm like, no, you can generate infinite synthetic data if you have a good judge.
But that's also that's what you need data for, judge.
Right.
You need to get good data to create a good judge, which is, yeah, you only need the data to create
good jobs.
Once you've got a good judge, you don't really need to.
Yeah, yeah, yeah.
So that's what like you hear from the labs as well.
Like a lot of them are doing synthetic data.
Now I have two questions.
One, how do you productionize this.
Do you export like the prompt and you run that.
Yeah.
So I don't productionize this.
Like I've never put a DSP program into production.
Exactly.
Yeah.
I could never convince them to do that.
And I wouldn't want to.
I don't want that because it's like a kind of complicated.
And yeah, maybe I'm warming to it.
Like I actually have an internal tool that uses DSP end to end.
Yeah, okay.
But there is a way to get it running with the fast API.
And so, so one of the things that is important is that you can save the program locally.
And then that creates a folder with all the metadata and the pickle of the program.
If you share this folder with someone, they can just literally just do this.
They can just load the program and then it runs perfectly on that machine.
But then I export a prompt.
Is that something you do?
Yeah.
So this is where my special hack and this is like the.
That's what I would want.
I wrote I pasted Omar like insistently because he like really hates that, that people just want to
get the prompt out and it's not that easy because even in inspect element it shows the outputs and stuff
and it's hard to construct it programmatically.
So anyway, he was like, you could just do this.
And I'm like, okay, fine.
So you basically pass in to the chat adapter, which is what it uses underneath the hood, the format
function, and then you get the signature, the demos for all of the named predictors essentially.
So it's yeah, I don't don't worry about it too much.
Your eyes will bleed if you look at it.
But but basically if you just run this then it will like you'll have the full.
And this is the same like this is exactly the same.
It's exactly the same.
It's got the system message and it turns it into the user and assistant messages as well.
So and and for variables like how like I normally use liquid but like it uses something else, I assume,
or it doesn't use variables importantly, which is interesting.
So okay, so how do I custom.
So yeah.
So how do I use make use of variables or like how does it even work.
It just has instructions.
And then you have a user message.
And then it will follow up with whatever the reasoning and things is from the user message.
So how do I trigger this thing then.
Yeah, exactly.
So you can see here it doesn't like use variable like even this actually isn't a variable.
It's just a formatting.
Yeah.
And then all it all the only place it uses the variables are here.
It just puts in the inputs and the outputs.
Yeah.
And then and then when you generate so you can see that the user message is like just topic like this.
And then religious satire.
So what I have to do is the exact same format.
Yeah.
You'd have to use the same format.
Or you could use the display chat adapter if you wanted to.
Or I replace those things with liquid text, like real stuff.
Like how you'd normally pump.
Yeah.
You can also use a different chat adapter.
So you can also create your own chat adapters.
Like you can make it work automatically.
Convert it into your own format the way you like to do it.
If it's markdown or or whatever it is.
So you can see we've got the prompt out here.
We've got the this is just the messages stream essentially.
So you could actually just pass this into ChatGPT as well or into OpenAI.
So I have one one other question that says yeah, which is it's interesting to see that it's doing it's
few shot examples as assistant user assistant a user assistant.
Yeah.
Is there a way to be like I want the examples to be inside just the system prompt.
You just have to change the chat adapter.
So yeah.
And it's all open source.
So you can see what they're doing and you can just subclass it.
And which is quite nice.
But but yeah I've found actually that it works better to just put the few shot examples in the system
message.
It tends to follow it better.
Yeah.
So I don't fully agree with the way they've done this, but it is.
This is like the way that you're supposed to do it.
But like, again, it's one of those hacky things where like, I found that doing it the wrong way actually
sometimes works, works as well.
Yeah.
And once again, like, the nice thing is that you can still benefit from everything by just changing
the adapter.
And yeah, if you change the adapter and optimize with that, you can recompile the program and then
it will just change a few short examples.
It would change all of the output input.
It would change everything to just match your format, which is because the adapter is used when running
the optimization.
Exactly.
So every single time any prompt runs from any of the optimizer processes or processes, it will always
go through the adapter to the LM and back.
And one last question from me.
So I have memories specifically tied to users.
And I would inject memory into the context as well to personalize things more.
But how would memory work?
Or how would that work with an optimizer or you just leave it out completely.
Yeah.
So no, I yeah, I've thought about this a lot and and the way you do it is you give it as like a specific
field in DSP.
So they actually have a history field somewhere.
They have memory like there is a concept of memory.
Yeah.
And they've got a tutorial about memory.
But what I've used for those because we also have the same thing with with rally like we have the past
chat history.
Um, yeah.
Exactly.
So there's this specific DSP history, uh, which will then format it in the correct way as a message
stream.
Yeah.
Okay.
But then but how do you like, like we have 5000 users.
Like how is the way you've got to think about it is it's like a, it's like a Excel sheet.
Right.
For every interaction with the user, you have the same as long as you have the same columns like the
input from the user is this.
And then the memory of the user is this.
And then maybe you did some rag right.
So like the rag result is this column, you know, and you can actually build the rag into the program
as well.
So you can do yeah, you can do all that stuff you can have.
Actually I think they've got an example here.
Yeah, yeah.
Here's an example.
Oh no.
Sorry.
Yeah.
Rag.
So like here they've used Colbert as the rag thing and they've just passed that in to search Wikipedia.
Right.
Or they have agents where it's like they have evaluate math and search Wikipedia and then they pass
them as the tools.
Right.
You can actually build whatever program it is you want, but you just need to make sure your examples
contain the user ID or something like that.
And then it can do whatever memory lookup for that user, and then it's included in the optimization.
Exactly.
Yeah.
And so that that's nice because it will make it extra generalizable in terms of it will because the
memories are different every time.
Like it will, I think, really, truly learn the task, if that makes sense.
Yeah, yeah.
But like every row is just a user interaction and what context interaction had at the time.
Yeah, yeah.
And you just include it or do it however you want.
Yeah.
exactly.
Yeah.
So I have a question.
If you have the time.
Yeah, sure.
So you say you don't use DSP in production.
How do you go from the prompt that you have in production to DSP.
Yeah.
Good question.
Yeah.
So typically what I'll do is I'll run the optimization.
So there's a few things that I tend to use it for.
One is to just script to try different things in different models.
So I'll be like okay does this work right.
Does like I don't know, I was trying to make like a focus group where the agents talk to each other
and it was much, much easier to just give them a talk tool and then make a react agent in DSP and then
let it run and just see if it worked.
So then it just gives me a broad strokes of does this interaction format work or not?
And it's crappy.
So I was like, okay, I'll abandon that.
It's really good for that.
I use it to test out theories of what might work, and then if it works in DSP, then I'll go into my
code and I'll test it.
The other way is I'll actually run the optimization.
I'll create the eval metric.
But then but then I'll export the prompt like you saw here, like I'll get the system message, etc.
and then I'll just translate that to my format.
So I'll just say, here's how I structure my prompt.
It's not perfect because it might change actually the evaluation score when you change the format.
But it's actually pretty easy to just give this to ChatGPT like paste it in and say, hey, I got this
program from DSP, can you just change the few shot examples?
So how do you go the other way?
Like for example, I have my prompt for spiral.
Let's say I have a very fuzzy class that I want to evaluate against.
How would I take that and then put it into DSP to then evaluate against.
Yeah.
So the canonical way for DSP would they would say, don't worry about all the crazy stuff that you put
in your prompt.
Just describe the task as the instructions, like just a simple task.
Describe what the inputs and outputs are and then dump your your eval data set and then create the focus
on the eval metric.
I found pretty bad results doing that.
Like it takes a bit of time to get back up to.
If you've got a really lovingly handcrafted prompt, it actually takes quite a while.
Like I actually have a joke prompt which is not safe for work because it's like very good doing Dave
Chappelle.
Can I even train his voice?
Um, and I didn't really want to get sued, but it is like you, it literally passes the Turing test.
And this was back with before.
Like before, uh, and it's unbelievable.
So I still haven't created a DSP joke generator that beats that.
But that took me like days, like a lot of I just love comedy, so I put a lot of effort into it, basically.
So I would say quite often I use DSP for the things I don't care that much about, which is 8,090% of
my problems.
For rally, we have a really lovingly handcrafted prompt for querying the personas and for generating
the personas.
But we also have prompts for like generating the title.
And I'm like, I don't care about that.
Like, I just use DSP for that.
Or I have a prompt for classifying the diversity of thought in the.
And I don't even know what diversity of thought means really in a lot of cases.
So like I just use DSP for that.
And, and I put in into my data set cases where the users have complained and send them to me.
So that's how I think about it, is create your classifiers.
It's amazing at that.
Create your like utility tasks and despite it's like amazing for that.
And you don't need if you don't care that much about it.
But just like I writing like I still handwrite stuff like my every article is actually the only thing
really that I like write myself anymore because they're going to 100,000 people.
It's actually, like worthwhile to spend a day.
And it only takes me a day because, like, I don't when my life really be better if I like managed
to get that down to a couple of hours, but like, maybe the quality won't be as good.
Yeah.
So like the things you really care about, I think you can still beat DSP, but like I would say increasing
amount of things I just don't care about.
Yeah.
So if I have a categorization thing would be would it be better than hence doing it categorization or
you would say still think it's better?
It will be better.
If you hadn't, I would say I haven't met a classification task.
That DSP didn't completely blow out of the water.
Okay, so I should use it for Quora email classification.
Yeah, I think so.
Yeah, I think that would be like a really good place to start.
Yeah yeah yeah.
The only thing there is like, how do I train it for all thousands of users?
Because all users have different rules and different nuance like I.
The problem is dynamic, like my problem changes per user and I'm just worried that will mess like it
will generalize something that's really good.
Yeah, to some degree the people.
Yeah.
To some degree I guess like it.
If there is a coherent meta task, as in if most of your users have similar preferences, then it will
do a pretty good job of teasing out if there is some pattern.
Maybe that's what it will do a good job of.
Break it up into steps where it's like one is the like the generalized version, and then it will do
a secondary for things, but that's more costly.
So think about it.
Yeah.
Yeah exactly.
But what you could do if you have enough data per user is you could just have a DSP program per user,
right?
Like I've seen that with fine tuning.
That would be like say writer did a really good job of this.
They made their own AI models.
They had the base model which was really good at writing, but then they trained a custom model for
each because their enterprise clients, right?
They trained a custom model for each enterprise client on their brand values and their tone of voice.
Or you could have I see AI image companies do this right.
Like they have a custom law of per brand.
Yeah.
Yeah, I think that might be the key.
And I am going to try that.
Yeah, yeah.
Do you like how costly is it normally to run this.
Do you know.
I would say you basically don't really have to worry too much if you're using GPT Mini or Haiku or one
of the Gemini Flash, you know, like that's you're never going to cost yourself more than a couple
of bucks, even with the heavy optimizers, where it gets really dangerous and scary is when you're
actually when you're using anything anthropic like sonnet and and particularly like opus.
Right.
Like that.
That could be like a $200 mistake.
Or if you're using the big open AI models as well.
I actually did a big project for a client, which was 5000 AI personas, and we were just using for
oh, but I was I did this big optimization and and my credit card got declined because we'd spent like
$900 in an hour and I was like, oh, so I had to ring up my client.
I was like, do you have a credit card that I could use to continue?
Yeah, it's pretty funny.
Yeah.
This is like a really funny, really funny account, you know?
Uh, so that was that was.
Yeah.
Okay.
But yeah, because we couldn't afford any more tokens.
Again, very fun.
But yeah, my margins are not that big because my product is $15 per month, and I already spent around
two and a half dollars per month in inference.
Just so like a dollar is, is important.
Yeah, yeah, so is important.
So I would say that you might want to do like a more of a meta task.
Then if you can optimize the prompt once and it gets 50% better for all users is better.
Yeah.
Then then like, you can up those costs over, like, the whole use space.
Yeah.
I at least I need to experiment and see.
Is it worth a dollar or $2 once to run or.
Yeah, to some degree as well.
There is a real you can actually really bring down your actual costs with DSP.
Optimization is an investment.
But you I've seen it happen where like you can take like I would take GPT 4.5 because it's good at creative
writing.
And then I would use that as the teacher model, or I'd use that to generate a synthetic data set.
Yeah.
You just need to get almost as good, like 8,090% good is 4.5 for that task.
Yeah.
So like distillation I think is like the primary the classification and distillation are like the two
major brain.
I already own the cheapest.
Essentially I already use flash 2.0 for classification.
Yeah.
Flash is great.
Unbelievable.
I hope Google keeps subsidizing it.
Yeah.
Cool.
This is amazing.
Thank you so much.
Cool.
Yeah yeah yeah yeah yeah.
It'll be good to get the recording afterwards.
We could turn this into.
Maybe I don't have to write my next every column now.
So yeah, we should like.
Yeah.
It's like I'm going to I'm definitely going to try this out next week or weeks because this is exactly
the problem I'm trying to solve.
And I'm just lazy to work on the problem.
Yeah, it is like one of those things where actually similar to like coding.
Like when I first started using copilot, I was like checking every line and then now I'm like, oh,
just see if code can do it.
And yeah, we'll see.
I think Jess and I used to care a lot about my prompts, and now I'm like, ah, well, just see if
I can do it.
Yeah, yeah.
Yeah.
Cool, cool.
All right.
Thanks, guys.
Yeah.
Thank you.
All right.
Take care.
See you.