-: I am gonna teach you about prompt testing
and the good news is you don't have to know
how to code to do this.
Here you can see the outcome of a test.
This is variation A versus variation B.
And specifically I was trying to test something
that would increase the word length of blog posts.
Even if you ask it for quite a long blog post,
it doesn't listen all the time.
There's all sorts of tricks to getting it to listen.
And the trick that I tried here,
if you look into the data sheet, I specifically just said,
"Make it really long or I lose my job."
The reason why it works,
if you look for the emotion prompting paper,
is that large language models can actually understand
and can be enhanced by emotional stimuli.
And that's pretty interesting
because AI is like a simulation of a human brain,
if you think about it that way.
So it does make sense that they would react
in similar ways to the way that we work, right?
So that's the technical explanation to why this works.
All right, so what you can see here,
and I've just outlined these,
is that these responses are actually longer
but not consistently.
In some cases they're shorter.
In this case, we actually got
like a pretty short response back.
In a few other cases we've gotten below a thousand words.
The way I'm calculating word length here
is I'm just counting the result
of splitting the text by space.
You can see that this doesn't have to be really complicated.
Evals is a whole topic, right?
There's a lot that you can learn about evals
and most of them are geared towards programmers
who can set up like a eval framework.
But in this case, you know,
you just need something simple in Google Sheets,
so you just need to count the number of words.
We have an existing experiment
and we've tested adding it in all caps,
but we want to add another variation here.
And that should be as simple as just creating variation C.
And then we're gonna run it 10 times.
Just count 10 of these.
There we go. So we have 10.
And the hypothesis I have
is that if we take this,
the hypothesis I have is that if I specify a word length,
then it might listen to that, right?
We'll see, and this is an experiment, right?
Because we don't actually know
if it's capable of following word length at this point.
I'm just gonna call this Specify Word Length.
And I'm gonna take this existing prompt, the control,
and it's important that you start from the control each time
because you might be getting different results
based on the different changes that you made.
I'm just gonna paste in the template here,
get rid of these quote marks.
And then I'm gonna say an outline for a blog post.
Each section should be a minimum of two paragraphs long,
and the overall article
should be more than 2000 words.
I'm gonna pass that in there.
And then, so this is all, you know,
this is gonna be the same every time.
And the actual post here,
I'm just gonna paste this in.
So this is when the template's filled in.
And if you see here, we have these in curly brackets,
we have writing style, we have topic,
these are variables that you need to fill in.
And if we see an example here, this is where it's filled in.
So you can see that we filled in the topic,
productivity with time blocking,
and we filled in the writing style that we want
and we filled in the example post as well.
So we've given it one example already,
which is a post here about the Pomodoro technique, right?
And then we're just gonna wait for it to come back
with the actual response.
We need to adapt this filled in.
We call it the formatted prompt as well,
because that's what we're actually gonna
copy and paste into ChatGPT.
So now we have it in here.
We need to just make that change
that we made to the template.
We said minimum of two paragraphs long
and it should be more than 2000 words.
Okay, so that's the only change that we're testing here,
and we're just isolating that one variable
and we can just copy and paste this down further, right?
So if you're running a more advanced experiment,
you might want to try multiple things.
So maybe you try multiple topics,
maybe you try different examples,
whatever it is, you can test that.
And the reason why, by the way,
I'm running the same thing 10 times,
is you can see just how often the results vary.
And this is the thing with LLMs,
they don't always give you the same response back.
So it's really important that you actually
take a look at what's coming back
and you actually run experiments like this.
Okay, cool, now we have, just to summarize,
we have the specified word length.
We have a hypothesis of what we think might work.
We have the actual prompt template,
which we're recording here for posterity,
and we could use that template to fill in different topics
or different writing styles, whatever it is.
Then we have the formatted prompt for this run, right?
And this is, each one of these rows is a run of ChatGPT.
And this is exact text that we've pasted in.
We will paste into ChatGPT,
so it's the prompt with the variables filled in.
That's the formatted prompt.
And then we just need to put in here
which model we're gonna use.
In this case we're gonna use GTP-4, which is ChatGPT+.
And then this is where we're gonna paste in the response.
We can also copy this formula down
and I'm just gonna drag that down there.
So that's gonna count the number of words.
Cool, all right.
So now I'm not gonna use ChatGPT
because ChatGPT I've found is relatively inconsistent.
It's somewhat personalized to people's results.
If you have customer instructions,
it's gonna change things, right?
I would say whenever you're templating something,
you want to use the API,
but obviously I promise you wouldn't have to code here,
so I'm not gonna make you,
but if you sign up for a developer account at OpenAI,
the main thing that I use here is this playground.
This is the system message.
This is like the customer instructions.
We're not gonna play with that here.
We're just gonna emulate talking to ChatGPT.
We have the model over here so we can use turbo preview,
we can use GPT-4 or whatever it is,
and then you can set the temperature as well.
I just leave it at one right now
and that's relatively balanced
if you want it to be more deterministic.
So you want it to give the same result every time
you could lower it or if you want it to be
a bit more crazy or creative, you could increase it.
The main thing you wanna make sure
is you set the maximum length here.
I'd set it for 4,000 because that should be enough.
One token, like it's saying here,
is roughly four characters of English text.
So about 3/4 of a word.
Yeah, 4,000 tokens would be about 3000 words.
It should be plenty for what we need.
And then don't worry about these other settings.
I'm gonna paste in the prompt here.
And we can see, actually, lemme just delete the quote marks,
this is something that Google Sheets adds.
But yeah, this is our formatted prompt.
Remember, don't place the template in.
So we have here the productivity with time blocking topic.
We have the writing style,
and then we have this example post of another post
I've written in the past.
And it's gonna go off that.
Okay, cool. So let's hit submit.
And it's gonna generate this response.
And this takes a little bit of time with GPT-4,
is much faster with GPT-3.5. (laughs)
But the nice thing about this, by the way,
is you're also not gonna hit your limits
because GPT-4, you can only send 50 messages a day
or in three hours, which is super annoying, right?
And if you have this situation
where you're trying to test prompt,
I need to run it 10 times,
that means I can only test five things and that sucks.
It's better to do it through
the OpenAI API playground instead.
One thing I should also mention
is that there is some cost to this.
GTP-4 is pretty cheap, but you can see the pricing
and the usage in here.
I'm just gonna try and open that in another tab.
So you can see I only spent three bucks this month,
but last month I spent, yeah, a fair bit, 130 bucks.
Just depends on what you're doing.
This actual, these API calls,
so like this one I just made here is like 5 cents.
So it is not a big amount, so don't worry too much about it.
Cool. All right, here we go.
We have response here
and I'm just gonna copy.
We're not even gonna read it.
I'm just gonna paste in here.
And here we go.
We've only got 933 words, so it did ignore
our admonition to make the word length long.
But you can have a couple of choices here.
Either one, you could iterate on the prompt
and just say, "It did so badly on the first try
that I'm not even gonna try it multiple times.
I'm gonna try something else."
Or you could keep running it 10 more times
and pasting these in just to see
if there's some variation.
I'm gonna do that and then I'll restart.
Okay, on the last run now, that you can see
that I've just been copy and pasting in here
and it's not looking good,
actually we're getting worse results so far.
Yeah, with an experiment like this,
typically I would run it a couple of times
and if it's not even close I would stop it.
But this is interesting data, right?
Because here we go, we've got the last one here,
if we just paste that one in.
What this tells us is that it does not respect word count.
It has basically completely ignored what we said here,
that it should be more than 2000 words.
That is a good result When it comes to an experiment,
you'd be surprised
because one, it tells us that technique doesn't work.
So there's a lot of time and effort,
like worrying about whether it's gonna work or not.
We're not gonna try that one next.
We're gonna try a different technique.
The other thing that's really helpful here
is that when we see other people using this technique,
we can tell them of this experiment
and actually give them the data
and say, look, it doesn't look like it works,
but you test it and it fosters a culture
of testing within the organization.
The really nice thing as well
is that this is just a small scale test
with you doing it in Google Sheets.
But like you could pass this to a developer
and say, "Hey, can you test this at a larger scale?"
Or maybe we could, if you find a result that does work
like we did with make it really long in caps,
the emotion prompting, then that's something
that you could validate with a larger scale test.
And you could try it with lots of different topics
or lots of different example blog posts
or lots of different writing styles
and see which of those things impacts the overall length.
You could also then scale this up
and do other types of evaluations.
So maybe the word length isn't as good
when we use one type of prompt,
but maybe the quality is high.
So you could have maybe outsource this,
have someone read each of these blog posts
and then put like a rating out of 10.
The reason why it's useful to save this stuff here
is that rather than just running it in ChatGPT,
and then you lose the output, you can never find it again
'cause it's in your history somewhere.
Then you can actually give this data to someone else.
You actually have some real data now.
And you can use that for whatever it is you need.
You can go back and review it in different ways,
or you can add new tests to the experiment.
Really, whatever it is you want,
that's a really powerful thing you can do here.
And just on the final thing,
let me just update this pivot table.
Okay, so to change the range,
you just select the pivot table, then you click Edit here,
and then if you scroll up, you'll see the range here.
Right now it goes to G21, which is just the bottom here.
And we want it to go all the way down to G31.
So we're just gonna change that.
Or if you want to make this automatic,
you could just say to G
and then it's gonna grab everything if you want,
but I'm just gonna do this manually.
Cool, all right, so we can see
that we actually had a lower response
then I'll lose my jobs in CAPS,
but we did have a slightly higher response to the control.
The way we figure that out,
I just copy and paste the formula.
Actually, lemme just do it here.
So it would be this divided by this minus one.
So it is 7.4% better in terms of length
across 10 observations.
So it's certainly pretty small,
but it did make a difference.
It just didn't make as much of a difference
as the all caps emotion response.
You could also test other things here,
so you could take away the caps
and just use the root of emotion.
Or you could put the word length in caps
as well if you want.
You can get pretty fine grain with this,
but now you know how to do prompt testing.
You can test whatever you want.
So feel free to make a copy of this sheet,
but also do make your own sheets
or add your own stuff if you find that necessary
for the purposes of your test
or the type of evidence you need to be able to give
to the other people on your team.