All right.
Let's talk about advanced optimization.
So once you have a prompt working what do you do in production to improve it.
And just as a reminder we've been going through the five principles of prompting in the previous section.
So if you haven't done that so far, it might be worth going back through the previous tutorial.
Now we have this social media task where it takes the insight, a social network, and then provides
a social post.
And just to show you how much we improved, we went from this full of emojis and hashtags and not that
great all the way through applying the different five principles.
And the latest result that we got was this we have five to pick from and they're ranked in terms of
rating.
The rating is done by an LLM prompt as well.
And we have, I think, a pretty good social media post from this.
Now we have something working.
You want to move away from basic evaluation and vibe checking, and you want to start to be more consistent
in terms of the things that you test.
The very first thing that I do, and I would say this is probably 80% of my work is run a B tests.
So I have a function here and it could be very simple.
You just generate for task A, you generate for task B, and each one is a different prompt.
And once you get the results out at the end, then you know which prompt is working better, and you
try to test 1 or 2 things at a time just to see what the results are.
Now, the reason why I have number of runs equals ten is that you need to run each task multiple times,
because otherwise you don't know how often it actually succeeds or fails.
We have this a b test prompts and then we have the generate and evaluate post function.
Here I'm just going to show you that.
So generate and evaluate.
Post first gets the response and then it gets the post content.
And then it evaluates the post content.
So it will come back with a rating here and the post content as well.
And that's what we're doing every time we do this.
So we're going to do that ten times for task A and ten times for task B.
Now let's compare the different prompts.
So the only thing I changed about the prompt from the previous example is that instead of in the style
of Malcolm Gladwell, we now say in the style of Malcolm Tucker, and he's a bit funnier.
He's a character from The Thick of It.
He's a bit sweary, actually, so we'll see if we get it to swear.
But that's literally the only thing we changed is the word.
Instead of Malcolm Gladwell, we changed it to Malcolm Tucker.
And when we run this, we're going to get a sense of over, in this case, 30 runs.
What is the average rating for Malcolm Tucker versus Malcolm Gladwell?
And this is doing all the heavy lifting for us, which is really nice.
We're going to get back the results B and we can compare them.
And this is a very simple pattern, but it's something that I do all the time.
Quite often I'll dump the results into a data frame.
So we have all the past examples and how often it did.
I'll also tend to test different examples.
Context.
In this case I'm just keeping it the same.
I have the same insight, but I never run it 30 times.
But you might want to run it say ten times, but then run it across five different insights just to
see what sort of edge cases does it fail on.
But in general, this is most of the work I'm doing now.
The interesting thing is I have a pretty simple change here.
Malcolm Tucker is Malcolm Gladwell, but you can make a much bigger change.
You could apply a whole technique, right?
If you're using a specific strategy, you have a hypothesis for what technique could work, then you
could make a change there.
Now here we go.
So we had a rating of 4.2 for Malcolm Gladwell.
And we have 4.3 on average for Malcolm Tucker.
And that's over 30.
So 60 responses there.
So it's pretty robust.
And you can see it just making a small change.
Now I have some proof that I could give to my client or just proof for myself if this is something I'm
automating for myself that the Malcolm Tucker style works and you could even go through and you could
test prompt C, prompt D, you could try different, different styles.
You could try different providing different examples.
And the examples partial.
You could apply different instructions in here.
There's so many different things you can do.
You know a different framework.
Instead of bait hook reward you could ask it to use a different social media framework.
And you change this section here so you can really do whatever you need.
And there are infinite different combinations of prompts that you could change in order to get to the
different evaluation score at the end.
You can also optimize your evaluation prompt as well.
That's not immune from a B testing as well.
All right.
So that's obviously a lot of work.
And one thing I have found which is particularly good for certain types of classifier is DSP.
So this is doing the above, but automatically for you.
And you have to do some setup.
So you have to set up the social media post generator here that inherits from DSP signature.
The signature is just where you define the inputs and outputs.
So we have the insight input field.
We have the social network input field.
And then we have the post that comes out at the end.
And this is actually the prompt that gets fed into DSP.
So I haven't written my big old prompt here.
I'm just going to see how DSP does with this.
Then we're generating the post.
We pass in the social media generator to a chain of thought tactic.
So this is implementing chain of thought for you.
And then we have to set up an evaluation metric.
The way you set up evaluation metrics in DSP is you.
The convention is you take in a gold answer, which is the correct answer and the context.
And then you take in a prediction which is the LLM answer, and then you have this trace equals none,
which is just required a break.
And what I'm doing here is I'm just finding the rating and then getting the score and passing it back.
I have to define which which model you're using and configure that.
And then you can just hit enter and it will actually it would actually generate this using this prompt.
It would generate a social media post.
And you can see here that this is actually pretty good.
This is without optimization.
So we haven't used DSP to optimize anything.
It was just taken this it's taken the data structure.
You passed it.
It's taken the chain of thought technique.
It's just brought all that together to make a pretty decent prompt for you.
And we're getting fairly good results then where the actual optimization comes through is in this bootstrap
few shot with random search.
And that sounds complicated, but literally all it means is it will add examples to the prompt in order
to improve the performance.
So you need to pass it some examples.
This is a training data set and the development set.
Then here I've just split the previous context and the previous examples.
The ones I ran 30 times for my a B test.
I've just taken them and I've split them into these two data sets.
You just need to make sure it's a DSP example format.
But and then you have to say what the inputs are.
So with inputs insight and social network, once you've done that you can just, you know, compile
it.
And literally this is the way you do it.
Optimizer dot compile and you pass in the train set and the value set.
And then at the end it will have optimized the the prompt.
So not the specific instructions but the examples that it includes.
And the reason it's called bootstrap few-shot is that it will include your examples, the ones that
you have, but it will also create synthetic examples as well and add them, meaning get the LLM to
come up with good examples to add to the prompt.
When you inspect a history here you can see this is the prompt that it came up with.
It's following it saying follow this format.
And then it's giving a bunch of examples here.
And you can see you can see the ones that it's added to the prompt.
It's added quite a few.
And some of them are ones that I gave.
And then some of them are ones that it's come up with, if that makes sense.
Listen up, you Muppets.
That's pretty interesting.
And then the final one, which is usually what I try last because it can be end up being expensive and
a little bit inflexible is fine tuning.
So fine tuning is really what you would do quite often to get a smaller open source model to perform
better or perform as good as GPT four.
But in this case, we're fine tuning GPT 3.5 and it's a pretty straightforward API.
I would I would say that you should try fine tuning the smaller GPT models through OpenAI first, because
if you can't get it working with them, then it probably won't work with the open source models either.
So I'd like to test I like to test out the viability of fine tuning using this type of example first.
Now all you really need to do for fine tuning, which is nice, is that you need to get the data into
the right format.
For OpenAI, it's just the messages, the messages list, and I'm just appending that to the fine tuning
data, and I'm saving it.
So you can see here that here's the list of messages.
And you have raw user content.
And then you have like raw assistant content.
So you get the user and the assistant responses.
And then you just create a you just upload the file.
So it needs to have the file in there.
Then once the file is uploaded then you can create a job with that file ID.
And right now you can't train, but you can train GPT 3.5 turbo.
Once that's created, then you have the job ID you'll be able to see that in the interface as well.
And once it's done and you can actually check this programmatically, then it would say when it's finished
at and it would give you the name of the model.
And it also says status equals succeeded.
Or if there's an error then you'll see what the error is and hopefully try and fix it.
That is how you set up and create a job.
And then how you actually use the model is like this.
I'm just going to hit run here just to see, okay, I need to create a job first.
So that's how you create a job.
Yeah.
So it's created a job there.
And then it's going to cost some money to fine tune.
And you can actually change some of the parameters as well.
But I just use the defaults.
But then once you've run the here we go, copy this fine tuned model here.
And I'm just going to change this.
So we're using the fine tuned model.
So in here this is looking for the status and seeing if it's working.
But I'm just going to swap in this fine tune model here.
So if you know the specific name of the model you're going to use and they're going to comment this
out here just because that's this one's still going to be running, right.
Actually you can see what the what the status is here that's still running.
But we're going to comment out this stuff which would check if it's running.
And if it's succeeded then it would set the fine tuned model.
And then we're just hardcoding the model name here.
And we're just going to see what this looks like.
So here I'm comparing GPT 3.5 turbo, the base model with GPT four, and then with the fine tuned model
as well.
So we're going to see which models perform best.
And we're going to see the difference here between what we're getting from the base model and what we're
getting from GPT four, and how close the fine tuned model is to GPT four, because I think that's really
the key, is get a lot of examples from GPT four and then distill those examples into a base model.
So this is just evaluating GPT 3.5 turbo ten times.
And then it should evaluate GPT 410 times and then fine tune model ten times as well.
I'm just going to uncomment this just in case you want to use this in the future, you should be able
to just run that and when your job is done then it will update the fine tuned model here.
So you say last job dot fine tune model.
But I already knew what the model was called which is why I hardcoded it there.
Okay, so let's finished evaluating GPT 3.5 turbo.
So evaluating it probably would have made sense to do this asynchronously because it would take less
time.
But being lazy for this case, the other benefit by the way of doing fine tuning is that you can sometimes
get better latency.
Because GPT four is a much bigger model.
You can see how long it's taking.
It's taking eight seconds per iteration versus two seconds per iteration for GPT 3.5.
So if you can get GPT four level responses from a smaller model, then that's much better for for your
application.
Okay, so it looks like we finished evaluating GPT four, which is now waiting for the fine tuned model
to see how that does.
Okay, now we're evaluating the fine tuned model that we created.
And this is only available in my organization.
So if you called this and would say you don't have access to this model, but the downside of it being
an OpenAI is that you also don't have access to this model.
You can't export it anywhere or run it locally.
You can see the latency is much better than GPT four, right, eight seconds versus 2.8 seconds, But
it is higher, unfortunately, than, you know, the base model.
So that's something to think about.
Now it's just running those evaluations.
So it's it's generated the ten but it's just evaluating them.
And we're going to get a score in a second okay.
Here we go.
So GPT 3.5 turbo got a score of 3.8 on average.
Got a couple of fours in there which is good.
And then if we just keep going here.
So these are the top three posts.
Then we have GPT four got a 4.5 on average.
So it's a killer you know it's really good.
And then here we go.
This is our fine tune model.
We got 4.2 on average.
This is much better right 4.2 is really good.
It's a big performance uplift from 3.8.
It's not quite as good as GPT four but I'm willing to live with that.
And we can actually see what are the types of responses we got.
Yeah here we go.
Sure.
Let's cut to the chase.
Even Einstein couldn't remember where he put his socks without a nudge.
Smarter AI models still need the human touch or prompt engineer.
Just like how you get a gentle reminder from legal HR and management.
So this is actually really good and I'm pretty happy with this.
We've got a 4.0 here.
So this is this is something that's really helpful.
This is a successful case of fine tuning.
And now that this has been successful now I would want to potentially go through the effort of trying
it with an open source model and maybe hosting it myself, or hosting it on replicate or some other
model provider service.
Once you've done the proof of concept and the fine tuning works, then it's worth optimizing the process
and maybe trying out different parameters and things.
But yeah, I'd say this is a last resort because it's a lot more intensive in terms of cost and and
know how.
Because I'm not a fine tuning engineer.
I'm not a machine learning engineer.
My expertise is in prompting.
So I only play with fine tuning, not doing it a lot right in production.
And typically what I'm doing is proving the case for prompting, for fine tuning, building the data
set for fine tuning, and then handing that over to an AI engineer who would know what are the different
parameters to optimize.