-: I'm gonna show you how to use ChatGPT
to automate your prompting.
And that's crazy
because we're using prompting
to automate every other industry.
Why not, if PowerShell is good at all these other tasks
and why wouldn't it be good at prompt engineering as well?
So this is screenshot from a paper here
and they called it Automatic Prompt Engineer
and they basically used, I think it was ChatGPT
or GPT Three back then to basically infer
what the instructions were if they give example responses
and then if they specific instructions,
what would the instructions be.
And so it comes back with the log probabilities.
So these are the different potential responses
and then it basically then ranks them
and I found that it was a human level
in terms of promised engineering.
Then there've been a few other papers
that have come out since then
including one that found a better prompt
than let's think step by step.
It does pretty well there in terms of accuracy.
And then there's a few other ones as well.
If you remember there was going around the internet,
this whole like take a deep breath
and then think step by step prompt that people were sharing
that was also using
like an automatic prompt engineer option.
So anyway, this is like pretty interesting.
I think in some cases it might work
better than humans at prompt engineering.
What I found is at the very least,
it's very good at getting you
up to above average prompt straight away
without you having to think very much.
Let me show you, I have a couple of these
but I'll start with this one.
I put this together based on a lot of testing,
but what I found was useful
and the system prompt is to actually tell it
you're an expert in prompt engineering.
Previously that wasn't very helpful
but now it knows what prompt engineering is.
That's with the latest updates.
It's updated I think to April last year.
So now it knows what prompt engineering is
and previously this didn't work
but now it does and we just basically tell
that you'll be given a prompt template,
one or more test cases.
Your job is to optimize the prompt template
using prompt engineering best practices.
What I've done here is I've gone a little bit further
and given it the best practices
and I've told it what prompt engineering is
just to get it in the right part
of latent space without making sure it's not thinking
of some other prompt engineering.
And then these are the actual best practices
and these are ones that I came up with as part of my book
that I'm writing for O'Reilly Media.
So these are the kind of the first three principles.
I found that they're quite useful.
And then the other thing you have to put in here
is the standard stuff like respond only
with your optimized prompt and nothing else,
don't cheat by including test cases
and the prompter, that sort of thing.
The place I got this, well at least the basics
of this without the principles
is there's a tool called Prompts Royale
and I think that they reference another paper
that kind of uses this, I looked into their code,
which is open source on GitHub and saw
what was working for them for system analysis.
Feel free to take a look at that as well.
They've got a bit of a different approach to this.
Anyways, what I found is if I put in the prompt template
and what's different with mine as well
as I put like these actual variables in here,
which is I think really helpful for me at least.
And the way this would work is if I want a list
of product names for this product then
and then these are like some examples.
What I found with the test cases
is that you couldn't put them in JSON very easily
or you had to put it in more in a YAML format
or markdown format.
Here we have the product description a variable
in the template and we were just trying to give
kind of example values.
So here I've put like a shoe that can fit a foot size
and then the response would be these.
And I've just given this is one shot
so I've just given it one example.
Then the other thing that I found worked really well
was adding these criteria evaluation.
You can put whatever evaluation criteria you want in here.
It actually handles multiple if you want,
but I've just used the default which is,
is the submission helpful, insightful and appropriate?
That I got from LangChain,
That's the default evaluation criteria at LangChain
when using their evals.
I also reminded that prompt engineering
best practices should be used
and then I found it's very helpful to include
in here any variables,
and you could build this programmatically like I have this
for one of my libraries I use for testing locally
where it just automatically builds this prompt up for me
and adds them based on anything
that's in curly brackets here.
Cool, so respond only
with your optimized prompt template.
I'm just gonna hit submit, I found it only really works
with GTP Four and only works quite well
with temperature of one.
A temperature higher than that it starts to get a bit crazy.
Temperature lower, tends to get a bit boring
but that's usually what works for me.
Here we go, it's pretty smart
because you see here, imagine you're head
of a creative marketing team, tasks coming,
catching opinion names, the product is blank,
you're looking for unique and innovative names
akin to how brand names like Google, Nike or Tesla
the respective industry.
So it's used some examples here
and it hasn't given a structure in the prompt
but I think it's done a pretty good job
and we could take this, just gonna copy this
and then we're gonna set up another one
and then we're just gonna put this as a pair
of shoes that fit any foot size
that was the product we wanted.
And then I'm just gonna submit there
and it comes up with pretty good names, so there you go.
It's created a prompt.
Now I didn't stop there, we have another template
'cause I found that you don't always have
like test cases and sometimes you just want be able
to just do it just from a task specifically.
This one also follows something from Prompts Royale
but you can see that it's using very similar system message
and then we've still got the the different
best practices there.
But the difference here is
that we say the prompt template must take context
from the user, the form of relevant input variables
surrounded by curly brackets, IE this.
That was like the hardest part to get.
Once it does that, it does a pretty good job.
And and the other thing I had to kind of mention
is these placeholders should be labeled in the template
as they will be replaced the values.
'cause one thing I found that it would do
is here it say please take the product category
that's done correctly, that's labeled it.
But sometimes it would just say please take the,
and then product category and then it wouldn't label it.
So when we insert a product category in here,
it wouldn't work, right,
because it would be referencing product category down here
but that text wouldn't be in here anymore,
if that makes sense.
Just to show you an example there.
So I'm just gonna get rid of this
and then so if you put in product category, shoes
and then we're gonna say target audience.
(person typing)
Say.
and then a distinct feature is just say
extra comfortable and then it knows
because it can see that like the target audience is up here
so then it can infer from these examples
if that makes sense.
That was. And then here we've got some more stuff.
We have to repeat the product category here
in this template, put it in shoes
And then target audience say at home
and then this would
be cool.
So see how that works.
Okay, cozy comfort homemaker,
oh god, it's called it a homemaker.
That's pretty terrible.
But you get the idea.
This was a task to generate product names.
We could change whatever task we want here.
This could be rewrite some text
in the style of the author
and then see what template comes up with.
And you can see it's using all the different principles.
The really cool thing here is that it actually inserts
these variables in so it makes it a real template
and it does a pretty good job of doing that.
Cool, yeah, hopefully that will make sense.
There's a lot of prompt engineering principles
baked into this as well
and you can see that it does a good job
because we've given it a lot of handholding here
and I think I used to try and do this back
before the update and because I didn't really know
what prompt engineering was,
it wasn't particularly good at it.
But now it does because it's in the training data.
So this opens up, I think a lot
of prompt engineers
haven't really tried this again recently.
And this opens up quite a lot of stuff you can do.
You can also, you can go even further, like you can get it
to generate cases like these things that go in here.
You can get it to generate good examples.
You could also get it to rate the responses as well.
So that's something I've been working on
for my thumb library, which should be out soon.
All right, hopefully that was useful.