I'm just going to walk you through a use case for AI and how I optimize the prompt.
And the specific task was to write a PRD or a product requirements document.
So your product manager or you work with one, then this is a useful feature.
But either way, even if you don't you'll be able to see how we optimize a prompt, a real prompt that
you would use in the world.
So specifically this prompt, the PRD we want to write is a feature that allows passengers to contact
Lyft drivers directly to recover lost items.
The reason I chose that is because there is actually one of these out there in the world, so you can
compare it.
So this is the ChatGPT response and it's pretty good.
There are some issues with it though, but I won't dive into the details.
You can have a look at this a link in the presentation, but it's pretty comprehensive.
It's well formatted.
It's professionally written like you'd expect from AI.
It's pretty good.
It's also user centric, like you specifically are good at.
I think the AI in particular is good at making sure that we follow up with practices in terms of gearing
the feature towards the user experience, but there's a few things that are lacking compared to an actual
product manager.
One is there's not much creativity.
It doesn't really draw from experience or make any niche references which you would expect from a real
human.
There's also no real kind of attempt to morph it towards what the what worked within that organization.
So there's no it's not in the right style that you would need for a to get it approved, typically because
everyone is different.
Can't expect ChatGPT to know.
But then the other thing is hallucinations.
We don't want it to make up statistics and things like that.
How we're going to address all of these issues with our optimization.
So these are the five principles of prompting.
If you've been through the beginning of the course you would have seen these give directions specify
format provide examples equality and divide labor.
So I'm just going to walk you through these.
Apply each one to the prompt.
So the first thing is give direction.
It's really important to understand what type of style you want.
You want this in the style of a fortune 500 executive, which you know is full of corporate speak and
it doesn't pay as much, but it's longer.
And or do you want to say like in the style of Steve Jobs?
And it doesn't have to be a specific famous person, but you want to give it some direction in terms
of what type of style that you want, and you can see how big of a difference that makes.
The other thing is specify the format.
So one thing I found really helpful is to ask it to use placeholders.
In this case I asked it to choose insert or wherever there's a number.
So I went through and edited that in the prompt, which you'll see.
And that means that you just need to control and find the different placeholders.
And then you can go and fill in those statistics yourself.
So that's quite helpful.
The other thing is just provide examples.
That's actually one of the most important things.
I chose this task specifically because we do have an example of someone actually writing a PID for this
task.
And then I took that example and I asked ChatGPT to create another one like it.
So instead it's this one.
It's the same sort of format, but it's guided virtual tours with drone footage.
And for example, so I took the real PID and I asked ChatGPT to make a different version of the same
one, the same structure.
And then I insert that as a prompt in order to get back to the initial task.
The important thing, by the way, is that this needs to match the formatting that you've asked it to
do.
You can see here I swapped out one of the metrics with the insert placeholder.
Value and quality is really important at this point as well.
You want to check whether it's performing better or worse.
What I did is I rated this example versus the traditional ChatGPT example, the default example that
you get if you just ask it to do this task without any of the prompting.
And I kept doing that until my prompt performed better than the original.
You could also keep going and get it to where it performs better than the real human version as well.
And you can a B test this with another LLM like.
For example, I would just dump both in without any context and a new chat window and then ask it to
rate these two.
The other option is you could get a real human rated to rate them, which is more accurate.
And then the main thing that you'll find is that you get to a point where you can't optimize much more
without dividing the prompt up into different things.
So one of the common ways of doing this is to add like a thinking section, a chain of thought.
This is this is technique that we're talking about here, where we're adding an additional step before
we start the prompt, where we ask it to think through the solution first and then answer the prompts.
And that tends to improve, especially for reasoning tasks like this, where it needs to make some assumptions
and and figure out what would be the right thing to do.
So in the end, we ended up with a good prompt.
I just wanted to share with you This was the actual example here from Lyft.
This is a famous example.
You can find it online.
I've got a link in there, but you can see this is what a human example looks like.
And it has some kind of specific things that are good about it in terms of it's very brief and to the
point, which I like, but there's a few things if you want to improve.
And in particular, one thing I find in general with these is I can't really describe what's good about
this.
So quite often what I'll do is I'll dump that human example into an email.
Like in this case, I've dumped it into code, and I've asked it to describe the writing style of this
PID in a brief paragraph.
And then I've used that as the instructions for my prompt.
So that was like the give direction step.
Then I've also generated some test cases as well.
So this is where I came up with new ideas like in this case PID for a feature that allows Airbnb hosts
to offer guided virtual tours.
So that was this is how you can come up with good test examples or good examples to put into your prompt
itself.
And then then you can run the prompt again on on the new examples.
So first to run the code for the Airbnb hosts virtual tours.
Then we get a pretty good response.
You can see this is much closer to what we get from the human, except we've got the thinking action
at the beginning here, and we've got the inserts.
And then it's just a brief into the point, which is exactly what we wanted.
And it's a lot more creative as well.
I think it's it's much more close to what I would expect from a real person.
Cool.
Solved the creativity issue.
We kept the good formatting, professional, nice and user centric approach, but now it's much closer
to our culture, which is what we want.
There's this brief and to the point factual type response, and we also solved the hallucination problem
as well.
So here's the optimized prompt and I'll share that with you guys so you can take a look.
So here's the example with a synthetic example where we, you know generated a fake version of the Airbnb
hosts.
And then this is how we rated the prompts as well, and use the rating to inform how we should ab test
the previous prompts, the different techniques we used here.
These are the specific ones that we found useful.
One was raw prompting.
We asked it to behave as a product manager for a major tech company.
The style of bundling.
We talked about this where we took the human example and asked it what was the type of style this was
written in, and then used those instructions in our prompt.
I also use emotion prompting.
So I put in there my manager will fire me if you make up any statistics.
So that got us to more reliably use the insert method.
And then we added the example with a real human example we talked about.
And then we also did chainsaw which was the planning step.
These are the tactics that work.
Sometimes these don't work.
You've got to test each one of them.
There's hundreds of prompt engineering tactics, but it really just takes a lot of trial and error.
So hopefully that gives you a bit of an insight into how the work of prompt engineering is done.