-: All right guys, I wanna show you the evaluator
optimizer pattern.
So we're gonna be using something based off this code right
here, but this is like an agent pattern
and it's quite useful.
I use it fairly regularly for situations where the cost
of the task is not that important
and also the time it takes
to do the task is not that important.
But it is very important that you get the task done well.
What it's using is like an LLM judge
where it will generate the task,
send the solution,
and the LLM judge will tell it where it failed
or what's wrong with it.
And it'll keep going in the loop
until it reaches the maximum number of iterations
or until it says, "Yeah, you've passed."
So there's many different ways to implement this.
They have some good code here
and I've used that as a starting point,
but just adapted it to a specific use case that I had.
So just going to go into our notebook,
and we'll just code along.
One thing, I'm just gonna grab the standard code
that you use to get the OpenAI API key
in Google Notebooks.
You just need to like turn on the API key
that you already have in the secret.
So you can just add the secret here,
and this is the easiest way to do it.
And then if you just run that,
then you hopefully should have your OpenAI key.
Go into the OpenAI account
and get one there if you don't have it already.
All right, so I just need to upgrade OpenAI
to the latest version.
Just gonna run that. Okay?
And then we are going to
create a function called the LLM.
So say run LLM.
This is just like standard practice.
Typically you're doing a bunch of this stuff.
So just gonna call this a user prompt.
It's gonna be a string. We need to pass the model.
It's gonna be a string.
We're gonna do gt4,
on average it can be the model
we're gonna be using the most.
And then we have a system prompt,
also string, and a default is none.
Alright, underneath here we're gonna have our messages.
We're gonna have if system prompt,
going to, yeah, add that.
It's now called developer by the way.
So we're gonna change that.
And yeah, we want to add the user prompt in here.
That makes sense. And then we do completion.
Okay, so it's got the old version here.
We need to get a client.
So let me just paste that in here.
And then we're gonna say client.chatcompletions.
Here we go. So now it's found the right thing here.
All right, we're gonna pass in model and messages.
Okay, I think we're good.
Now let's just check that this is working.
Yeah, what is the meaning of life?
Here we go.
Hopefully, it would say 42,
but yeah, here we are.
Okay and then we need to set up our generator
and discriminator or evaluator and optimizer
or infrastructure.
So let's have the task.
This is like the first thing we need.
So the task is one that I was using the other day.
Write a one sentence bedtime story.
(keyboard clacking)
about a unicorn for a five-year old girl.
I was doing that for my daughter.
Always good to automate your life.
And we're gonna do a generator prompt.
I'm just gonna paste this one in because it's a bit longer.
I'll walk you through it.
So your goal is to complete the task based on user input.
If there are feedback from your previous generations,
you should reflect on them to improve your solution.
I'll put your answer concisely in this format.
Thoughts and then response.
And we're taking that from the code in here.
So they have this generator prompt here.
We're also gonna get the evaluator prompt.
I'm gonna paste that one in here as well.
I've changed this one a little bit though,
so I'll walk you through it.
Evaluate this final response four.
And then you have your criteria.
These are the things that you want to get out of this task.
And it's gonna keep going until it gets to this criteria.
So one, I want it to be age appropriate.
I want it to follow style
and best practices for like children's story,
but I also want it to be only 10 words or fewer.
And this is gonna be a hard thing for it to do
because it tends to be a bit verbose otherwise.
All right, so now let's set up our generate task.
Generate for the jerry task.
We need a string as the task.
Then we need our generator prompt.
So string we'll need some context.
Also, string default is just empty string.
And what this is gonna return
is a tuple and two strings.
Okay, so we're gonna say full prompt.
And the full prompt is gonna be this,
just gonna copy this across from the code.
We're gonna take the generator prompt, put it in first,
then we're gonna add the context, then we'll add the task.
Then that's if there is context.
By the way, if there isn't context,
then it would just be generate a prompt and task.
All right, so hopefully that makes sense for you.
Just gonna give this a comment as well,
just so of the doc string.
And then we just need to run that response.
So run the response, print out, generation start
and the output and then return the response.
Okay now we just need to do the same thing for evaluate,
which is a bit more complicated.
And so we have our evaluator prompt.
Let's do our evaluate function.
So the evaluate needs task,
it also needs the evaluator prompt
and we need the generators content.
Okay and so far so good.
Also gonna return the tuple strings.
And here we just need to say,
evaluate if a solution meets requirements,
because that's what we're gonna try and do.
All right, first we need to get the full prompt.
I'm just gonna paste this across rather than typing it all.
So we're gonna take the evaluator prompt,
and then we're gonna put the original task
and then the content to evaluate.
So it has all the full context of what it's looking at.
And then we're just gonna run it right, run the LLM,
get the full prompt and we need to do some regex.
So in the original one, they used pedantic
and they did structured outputs.
I'm not trying to teach that right now.
So we're gonna use regex.
That means we need to do import re.
This is like the dumb version.
So all I'm doing here is I'm getting the status
and I'm getting the feedback
because it's gonna send it back in this format.
We're gonna get the status
and then we're gonna get the feedback.
And I just asked ChatGPT to write this for me.
You can ask Gemini in here if you want as well.
LMS are pretty good at regex.
If there's an error, we need to raise that error
and just say it didn't come back with a response.
So we'll be able to see that.
But then if we do get the match,
then we're gonna get the evaluation
and then we're gonna get the feedback
and we'll just print them out and then return them.
So that should be good. Alright, cool.
So now we have everything we need
and we just need to write the loop.
Okay, so loop workflow,
this is where we tie everything together.
So this is like our master orchestration.
Starting to do this for us now, right?
So we have, 'cause we've been adding type hints
and stuff along the way, so now it knows what we're doing.
Tasks, string, generator prompt is a string.
Evaluator prompt is a string
and generator prompt is a string,
and then it's actually returning.
We're gonna return a tuple, which is a string,
and there's a list of dictionaries.
All right, what do we need to do in this function?
We need to keep generating and evaluating
until it passes the last generated response.
So first we need a marry.
It's just gonna be an array
and then we're gonna generate a response
and add it to memory, right?
So far so good. Then we're gonna go into a loop.
So we're gonna do a maximum number of iterations.
You could put this in parameters
or I'd just like to hard code it.
And we're gonna say while max iterations
is bigger than zero, then start doing some stuff.
So this will mean it will keep looping through.
And then at the bottom we're gonna say max iterations
minus equals one.
So that way it's gonna start at five
and then go to 4, 3, 2, 1 and then stop.
It's really important with agents
that you don't just let it keep going forever
because it might cost you a lot.
That's the reason why we do that. Okay?
And then we need to get the evaluation,
the feedback back from evaluate
because that's what it sends back here.
And we just need to send in the task, the evaluator prompt
and the response.
So that's pretty straightforward.
And then to terminate the condition, we're gonna say,
let's say upper if, if we get pass back instead of fail,
then we're just gonna return the response okay?
So that means everything is gonna be done, right?
So we're gonna get exit out of this function
and that doesn't matter if,
it's not gonna do all five iterations if we don't need to,
but if we, presumably we didn't pass;
then we need to join all the previous attempts together.
So this is just iterating through the memory,
getting the feedback, and just joining all that together.
So you're gonna see that once we run it
and we're gonna run the response again.
So this is, we could do chat session management,
we haven't done it in this way,
but this is like an easy way to do it,
where the loop itself understands the memory.
The reason why you might wanna do that, by the way,
rather than keeping the memory
within the individual messages streams is
that you can format it in whatever way you want,
which is quite nice.
Cool alright, so now let's try our loop workflow
and hopefully if we've done all this correct,
then it'll run.
So we need our evaluator prompt, generator prompt.
Okay, I think that's everything.
Hopefully we did again there.
All right. Okay, we did get an error.
Could not pass the response.
Did we get these the wrong way round?
Yeah.
Here we go. See if this works.
Okay, here we go. It's working.
So first we output the thoughts and then the response.
And you can see it's too long. So we failed.
The evaluation started, it said it failed.
It is appropriate for a 5-year-old,
but it's not 10 words are fewer
and it does follow best practices.
You can see it's going through,
it's in a couple of loops now.
Then look at this, it was much shorter on the second try,
which is cool.
But in this case, it's saying
that it's not following best practices.
Also it's saying there's still not 10 words or fewer.
Yeah, this one's a bit too long.
Let's see what we're at, where we're at.
This is much shorter.
Just taking in that information from the previous one.
Here we go. We've got a pass yay.
Luna, the unicorn soared,
leaving sparkling rainbows in her wake.
Perfect, right?
It is age appropriate.
It successfully captures magical
and engaging image using vivid imagery.
Yeah, it's doing well on all of our counts.
This is a really good example where
the evaluation criteria might not be obvious
in the prompt itself.
And in this case, for example, we didn't have in the prompt
that it should be 10 words or fewer.
But also generally LLMs, if we put, make this 10 words
or fewer or make this a thousand word blog post,
they tend to do a bad job of that.
So something like this can really help you.
It's actually much better at evaluating whether it
did a good job the first time,
then it would be at doing the job the second time.
Cool, yeah, this is really helpful.
Where you want to use this type of thing
because it is slower, right?
Like you have to wait for it to keep coming back and forth.
And it's also much more expensive
because now we've done, we're doing two calls per round
and we're doing five rounds, there's 10 calls,
whereas normally we just do one.
The benefit is that we're trading off time
and money for quality
because we can put whatever criteria we want
in this evaluation prompt,
and it's gonna pass them eventually,
so we can just throw money at the problem until it passes.
There's lots of different ways to implement it,
but it just depends on what makes sense
for your application.