We're going to talk about prompt optimization.
And this is one long Jupyter notebook.
But we're actually going to break this into two parts.
So the first one is the one that we're talking about right now, which is optimizing a prompt for production
by applying the five principles of prompting from our book.
So I'm going to run through those, and then I'm going to do a second one, which is going to be a follow
up to this, where we're going to talk about more advanced optimization techniques.
All right.
Let's get started.
So we have a task and you always have a task with AI, right?
There's always something you need to automate or something that your tool needs to do like a feature.
And usually I write it in this format.
What are the inputs and what are the outputs?
In this case, it's a simple task.
You need to input an insight that I have about the way the world works, and I need to input a social
network like LinkedIn or Twitter.
And then I need I'll get a social post at the end of it.
That's ultimately what this does.
And by being really clear on what the insights are and really clear on what the outputs are expected
to be, you can already do a pretty good job with prompt optimization.
We're just going to write a really simple template.
It's going to take both of the inputs, write the social media post about how insight for social network.
And I'm just going to run that just so we see I we're running that.
Just look at the insight that I'm providing here.
Prompt engineering will still be needed with smarter models, even as genius humans need prompting.
Even genius humans need prompting from Lego, HR management, etc. to align with business interests.
So this is something that I believe about the world, and it is not a belief that everyone shares.
And the reason why you need this, by the way, is yeah, I could just say, right.
A social media post about topic, but it's not going to be very good.
You tend to need to provide the secret sauce to the AI, and this is something that I've learned, and
therefore it's going to write a much better social post when I provide it that insight.
And let's see what we got.
Now.
The outputs not great because it's got these silly emojis and hashtags, but the underlying structure
is good as AI models get smarter, you think prompt engineer would fade away?
Think again.
Even the most brilliant minds need guidance from legal, HR and management, etc. it's already a good
start, but I don't really like the style.
So let's apply the five principles of prompting and get it ready for production.
The first principle is to give direction.
So describe a desired style or provide a relevant persona.
So I'm just going to run this.
And what I've done to change this is this is exactly the same prompt.
But now I've said in the style of Malcolm Gladwell, and specifically I've given it some instructions
that are really important for making the post sound authentically human and colloquial.
The response I'm getting here is much better.
Imagine a world where even the brightest minds think Einstein, Maya Angelou, Sherlock Holmes needed
occasional nudges.
So it's actually giving some interesting examples, but it still has these weird emojis and hashtags.
So let's keep going and keep improving performance and apply the second principle, which is to specify
the format.
And this is one where you could just mean the format of how it responds.
Do I want it in JSON or YAML or an ordered list or whatever it is, but also the frameworks that you're
using.
And I've specifically implemented a framework here called bait hook reward.
This is something that I use for my social posts when I'm writing them manually.
So I'm asking the AI to follow these, like what will grab the attention, what will keep their attention,
and then how do we reward them for paying attention?
And I'm asking them to write this out first in a kind of like chain of thought style.
So it's going to write in YAML, which is an easy to understand format.
It's going to give me the bait, the hook, the rod, and then it's going to write the post content.
So it's going to end up with a much better result.
And I'm still providing it with that insight.
And you can see here in the output it writes the bait, which is the even the smallest models need a
little nudge.
And then it's writing that hook and then the reward, which is what it can take away.
And then if finally brings everything together to this prompt.
So this is a much better result already.
We're giving a specific framework, and if you gave it a different framework, you might get a different
response as well.
All right.
Now let's go into the most impactful response which is providing examples.
This is the best principle.
It's also costly because you need to give it some examples.
And I've taken the exact same prompt by the way.
And then I've just added this example's partial.
I've kept this separate just so it's easier to change the examples in the future.
But essentially what I've done is manually written these.
So I went through the framework, I wrote a bait, I wrote a hook, I wrote a reward, and then I wrote
the post in my kind of copywriting style.
So this is really helpful, and I've given examples of how it would look for the Instagram or Facebook
or LinkedIn or Twitter as well.
So it should be able to learn from these examples and do a better job.
And you can see here that this is just adding the examples to the prompt.
This is the examples partial that we're adding in here we have the example and then we have the YAML.
And then we have the example two, example three and so on.
Now let's see how much better it does feeding in the examples partial now as a parameter as well as
just the social network and insight.
And I think it does a lot better.
We got rid of most of the emojis and hashtags because it's learned, you know, I didn't include them
in my examples and it's, I think, a much higher character in terms of the quality.
It feels more authentically human.
Here's the kicker.
Prompt engineering is still be vital, right?
That's pretty cool.
I really like that.
Now, what I want to get into now is the evaluation metrics, because you can see I'm still getting
these emojis.
So I could create a function to check for emojis here and then start to see like how many examples do
I need to bring in in order to make sure that there are no emojis found?
What's the emoji count?
And that's where you start to get into evaluating quality.
That's the fourth principle.
But emojis that's one one thing to check for, but it's not necessarily going to be the most important
thing.
I want to evaluate whether this is going to be engaging.
So I've given it a valuation prompt and this is usually what I do like.
Obviously you can have manual human responses.
You can programmatically check for performance of this function that checks for emojis.
But what I tend to find is that for most tasks, you need an LLM as a judge.
And in this case, I'm just asking it does it have grab attention, does it have a hook, what's the
reward, etc..
So all the things I expect from a social post and then I'm passing in that context.
What I'm asking it to do is to give the analysis and then the rating afterwards.
Because by writing the analysis first, you're going to get a more consistent rating afterwards.
And this is again using the chain of thought style response.
So let me just evaluate this on a basic output.
I'm just going to see what it gives me in terms of an engagement score.
So it's written all this information and then it said the rating is four.
All right.
Now that we have an evaluation prompt, that's usually when our cue is to start splitting things up.
And because most production systems are not just one prompt right.
Like it's actually multiple prompts put together, this is where we get into the principle of dividing
labor.
So I'm using async here here to run these things.
I'm just running asynchronously because I want to be able to generate multiple versions of this social
post and then get a rating at the end.
So I have this function generate a rank desk.
So generating rank descending and all it does is just run everything we had above.
but it's just doing it multiple times.
So here I'm generating five posts asynchronously all at the same time, and then I'm gathering the results
and then sorting them by rating.
So when we run this what we get is not just one prompt, but we have this evaluation step as well.
It's running the prompt and then it's running the second step which is evaluation which is here.
And and then it's getting the ratings and ranking them and maybe even three steps you could call this.
And what that does for us, which you can see is it allows us to filter out bad examples.
This one you got a rating of zero because I think specifically I said anything that has hallucinations
or made up statistics should be ranked zero so that we've avoided that bullet.
Right.
But then we have a few here that are rated five and then one that's rated four.
So now we have the very best one at the top, which is really helpful.
And we can make sure the quality is high.
Then when we have that split up into multiple sections.
So that's the thing that I tend to do to get things into production.
I would say now is the point where we have 1 or 2 evaluation metrics.
We have the test split up into a couple of prompts, and we have we're using best practices in terms
of giving direction, specifying the format or the framework to use, and then providing examples as
well.
This is probably all you need to do to get it working right.
This is this prompt is, I think, working pretty well.
And I would I would post this to my social social networks so that they're already in a really good
position.
And then once you're in production, that's when you moved on to more advanced optimization, which
we'll talk about next.