-: The fourth principle is where we start to move
outside the prompt itself and more into the wider system.
We wanna identify how often the prompt gets errors
and we wanna rate the responses
and test what drives performance.
So this isn't specifically a tactic to apply to the prompt,
but really a meta tactic
for getting better prompts in general.
Now, if we look at the text models,
we have no evaluation in this case.
We're just prompting ChatGPT.
Now, this is a simple prompt to get product names
that can fit any foot size.
Whereas when we're prompt engineering. we have the case...
Whereas when we're prompt engineering,
we really wanna focus on running the prompt multiple times
and we don't just wanna run it once,
like I've tested this template hundreds of times
just to see how often it fails
and then I've iterated on it since then.
So if we look at the prompt template,
you can see it's really well optimized
and there's nothing superfluous in there.
The specific place where you would evaluate quality
in ChatGPT is really just to rerun the prompt, say 10 times.
Now, I typically evaluate quality when I'm writing code
because much easier to run it 10 and 100 times
and then check the responses.
But in this case, you can actually just do it
in the ChatGPT interface as well.
You can see I've run this 10 times,
and I can see that it's providing.
It's providing good quality product names.
And in the run up to building this template,
I tested lots of different combinations of examples,
of descriptions, of direction, formatting,
and settled upon this as it works and most reliable.
Now, you can actually test lots of different things.
You know, I talked about reliability,
but one of the main things is quality,
like are the names actually any good?
Now this case I specified
I wanted something in the style of Steve Jobs
and you have Flex, iFit, and you have iShoe,
but OmniStep doesn't have the I in front of it.
So I found that sometimes this was like one in 10 results,
it doesn't include the I at the beginning.
And if that's really important to you,
then that's something you might wanna test for.
You can also look at the length.
So I have pretty short names here.
One of the main things I wanted it to do was to make...
Really long names are not gonna be as memorable.
You have iShoe, which is great, iFlex,
but then we have one example here is Adapt-a-Step iSneakers,
really got off the rails.
And that's something you can test programmatically.
You don't need a human to test your results.
You could just actually run a script that says
if the name is longer than a set number of characters,
then, you know, we've done a bad job.
And you can run the prompt 10 times, 100 times,
and see how often it creates one of these long names.
The other thing is format.
This is particularly important when you're coding
because the output of a prompt
is typically what you are gonna pass on
in the next step in the chain.
And in this case, it came up with a product description
that's part of the response
that didn't come up with the names.
It actually just came back with this whole description.
"Innovative and revolutionary,
these shoes are designed to adapt to any foot size."
It's not what I asked it to do. It's not part of the prompt.
And this is an example of what happened before.
I provided three different examples for it to go off.
Once you provide more examples,
the prompt gets a little bit more reliable
and this formatting issue doesn't happen as often.
This is something, again, you can test for in code.
If you're specifying JSON, you can pass the JSON
and check if that passing worked or not
before you decide to retry or stop that step.
Let's look at image models
and how evaluation of quality applies to that.
Again, we just have our normal prompts
where we're prompting Stable Diffusion.
We're trying to find a shoe that can fit any foot size
and the results are all over the map.
But the really nice thing about this type of interface
is that with image models,
quite often, evaluation is built into the platform.
So on Midjourney on Stable Diffusion,
you get multiple responses back
and you can choose which one you like.
Yeah, I would run this multiple times
and then when I find one I like,
you can actually grab the seed of that image
and apply that in the advanced settings,
so it's gonna be constraining the creativity
to just that type of shoe.
You can do a lot actually in terms of evaluation,
but the evaluation is usually human and manual.
It's very hard to program
programmatic evaluations or metrics.
You typically just have to look at the responses
and then click on the ones you liked and then save them
and then maybe use them as examples later.
That's the prompt template, if you did wanna use that.
But you can get similar results to me
if you use that template.
We can evaluate the quality that you get back.
Here, I've selected the one that I like
and then I can use that as the base image
when I'm generating small variations
because maybe I don't like some aspects of this.
I don't like how the lip is pointing up, for example,
but I do the rest of the shoe.
So if I select that seed image,
I could run it again multiple times if I want.
But either way, at the end of the day,
I'm getting a good idea of what are the options
and then I'm choosing which one is good.
Equally, Stable Diffusion by getting some feedback
in terms of here are the four that we've generated
and now the user has decided to download that one
or they've decided to upscale that one
to a higher resolution,
so that means that of these four,
the model did a good job with the one in the bottom,
so that helps them improve the quality of their models too.
Now, you can test different parameters with image models
and that's where I see a lot of experimentation.
There's something called the CFG Scale,
the Classifier-Free Guidance in Stable Diffusion.
And in DreamStudio, they call it prompt strength.
This is how much does the model follow your prompt
versus being creative in itself.
You can see the impact of this.
What I like to do is create an image grid
where I'm changing the parameter
and I can see visually the difference between them.
So you can see with CFG scale of three, which is low,
and then we're getting this kind of weird looking image,
it's not as good, it's not really following the prompt.
And then when we dial up the CFG scale,
we're getting much better results
and testing where in these parameters
the sweet spot is really,
you know, big key for image generation.