Right.
How do we optimize a prompt for production?
We're going to take this specific example, which is a news article summary prompt.
So here's an example.
I'm using Gene II the gene reader by the way in order to download this article.
Scrape it.
That's quite a nice easy hack.
You just put a r I in front of any URL and it should give you back the text in markdown, which is a
very useful format for putting it into an LLM.
But the actual prompt itself is literally just.
Please summarize the following user.
Now with any of these tasks, typically I'll break open a Jupyter notebook and then get OpenAI, which
is the main way that I use Llms is just directly through the OpenAI API, and I have just a basic function
for completion.
Here we're just putting in the prompt and then we're getting the context.
You're passing the context and then we're generating that.
By the way one thing I tend to do is I'll pass the prompt in here with Any variables in curly brackets.
And then I'll use prompt dot format and then I'll pass in.
I'll unbundle the context.
So the context is just a dictionary of different the variable.
And then what the value of that variable is.
And I find that that's quite useful for, for doing this.
All right.
So this is the naive response we get.
And I'm not going to read the whole thing.
It's an article about Donald Trump and just a relatively like politically biased article.
And don't mean to pass judgment either way.
I just mean that specifically the adversarial one to Donald Trump, which maybe he deserves.
I'm not going to weigh in, but but essentially, I wanted to take an article that was quite emotionally
charged and just see how well that shines through in the response.
And unfortunately, it doesn't really shine through enough.
We might lose some of the meaning of the original article, I think, if you're summarizing it with
a naive response.
So here's a more optimized response.
And this is after doing a little optimization.
And one of the things you'll notice is that this is a lot more making its reviews and its responses
like its much more biased and in the way that the original article is, which is what we wanted.
But it does have citations as well, which is also very useful.
It's looked up different things and I'll show you how we did that.
So we're applying the five principles of prompting here.
The first one is just give direction.
So the first thing I tried was I wanted it to preserve a nuance of the original author's intended tone.
So that's just the first thing that I would try and add in order to improve this prompt.
And it does do a much better job when you do that, which is quite helpful.
And then the second thing that I've started to do is specify the format, what do I actually want out
of this?
And one thing I've found quite useful is if I ask it to give me the key points or extract some information
first, and then it does a better summary afterwards.
And I've also asked it to bring me back this in JSON format, so I could use it in a program later on.
And you can see here it gives me the key points.
These are the things that kind of check.
And I found this is much better in debugging because it gets the wrong key points to summary.
It would be very good.
I find that's useful as well.
It's almost like a chain of thought type prompt as well, because if the key points are like the thinking
step and we have it before the actual summary, so it does improve the summary.
But the main thing you do to improve performance is just add examples.
So I went through I added some examples manually from other articles.
And yeah, one thing I should note here is that you don't always have to input the full thing.
So in this case I'm just showing the example outputs.
I didn't paste in the whole article text because then you're prompting are getting huge, and it doesn't
necessarily need that full article text in order to to learn from your examples.
This is what we get after we do that and it ends up being a lot more useful.
Again, I'm not going to bore you too much, but if you're interested in this, but I've just it is
a lot more close to the original article intent.
The fourth thing they tend to do is to try and evaluate the quality of the performance.
up until now, you just vibe checking and just what?
What you think it does well, but you really want to get a more formal definition as as quickly as possible.
So in this case, I used a function that you get the rouge scores, which is the the actual scores of
the the similarity between the original text and the summary which is commonly used.
It's just calculating it.
And then I have, uh, those rouge scores into a prompt and then asked ChatGPT to interpret them for
me.
So this is a typical response.
Compare the two rouge scores and then it will tell you what the winner is.
Overall, this is quite helpful because it means you don't have to interpret it yourself, and especially
useful when some of the scores are better and some are worse.
And so in this case, it's the overall chosen 1 or 2 is the winner.
But the first response was actually better on rouge one.
For example.
And then you want to break it into multiple steps.
So divide up the labor labour, essentially.
It's very hard to do all of these things in one prompt.
One of the steps I added here, which is something I found quite useful, is in some respects, I wanted
the summary to be matching the original intent of the article.
I don't want it to change the the style of the article.
So we've solved for that problem.
But I do want some fact checking here, right?
Like I want to see how biased in one direction it is.
So I added another prompt at the end where we would rewrite the summary, adding citations from the
fact check key points where necessary, and absolutely add 1 or 2.
But I got the fact checking from using Calveley, which is a search engine similar to perplexity.
So what it does is it will search out the course for these specific things, and then it will fill in
each of the key points that will fill in some solution.
It will say, yes, this is true or no, this isn't true.
And then some context.
And then I'm passing that into the prompt here with the fact checks section.
So this is the type of response we get.
You can see that it has the fact checks at the end, which is nice.
From there.
And that code is in the Jupyter notebook if you want to take a look at it.
There's a call a whole nother module on how to use tavli in the course.
So check that out.
Now once you get that prompt into production then this is usually good enough right.
But you want to keep improving it.
So now you have an evaluation metric.
You can run a B tests.
So you can test different versions of the prompt and see which ones get better scores.
You can also use the spy so you can do optimization instead of a B testing yourself.
You get the spy to write prompts for you, which is quite nice.
And then once you have a big enough data set of successful responses, then you might want to consider
fine tuning.
So usually I wouldn't start the spy until I've got 50 plus examples and then fine tuning.
Or maybe wait until I have 200 plus examples.
Or if the cost of the production is getting too expensive and I want to train like a small or an open
source model to do this task.
Hopefully that gives you a bit of a sense of what prompt engineers do day to day.