-: All right, I'm gonna walk you through a technique
that I call Personas of Thought.
I think that's just a play on the chain of thought idea
where you would get an LLMC,
then some tokens to think about the task first
before doing it.
And this adaption that I've been using more recently is
that you ask it to come up with a bunch
of different personalities or personas.
Personas are experts who will all have
a different opinion on the task,
and then you can aggregate their opinions together
to get the final result.
And I find that it's really good for the rest
of your thought and uniqueness of insights,
rather than just the generic bland stuff
that you get from an LLM.
So here's a code that I'm running here.
Just have a question that I want to ask.
Like I'm trying to come up with an idea
of whether I should use this product name
for a new product I invented,
and I just call OpenAI.
This is (indistinct),
and the prompt is very simple.
It's just I want a paragraph response
to the following question.
And the typical sycophantic stuff, it's like,
it's a strong choice, it's really good.
And yeah, it's catchy and easy to remember,
and I actually think this is a good name.
But yeah, this is a typical LLM response.
It's, you know, very rarely is useful information
if you just prompt it naively.
So the next one we're gonna try is the experts prompt.
I'm gonna run this, but this is very different.
I want a paragraph response to the following question.
But first, I want you to name a number
of world-class experts past or present
who would be great at answering this.
Then for each expert,
I want you to answer critically from their perspective,
given their background experience.
And then finally combine all of that together
into a single response, as if they'd collaborated
in writing a joint anonymous answer.
And I've given it some structure in terms of the format.
So we'll see what we get.
It came up with, in this case, a Nike designer,
the former FDA commissioner, an invent,
the Harvard Business School Professor,
Tim Ferriss, Tony Hsieh.
These are all fairly relevant, which is great,
and it's good at coming up
with good ideas here for inspiration,
and then as given individual feedback free.
Tinker Hatfield, thought it was good,
but this one, this guy David Kessler,
I was worried about that,
whether it resonates with consumers health needs,
and that makes sense
because he's an expert in consumer health.
So that's the sort of thing that just wouldn't come up
and didn't come up in the previous feedback
that we got from the LLM,
but because it's been pushed
to get a diverse crowd of opinions,
and then it ends up being a quite useful opinion.
So then you can go through
and you can look and see which feedback you agree with
and I find that that's really great
for brainstorming in general,
but then also we get this final response at the end
and it has some of that diversity of thought in here.
It says, "However, it's essential the name
is distinctive enough,"
and then, yeah, it doesn't specifically mention
the health benefits in this case,
but it's not very sycophantic,
like it does bring a bunch of these ideas together,
which is really helpful.
I find this technique works really well.
Now, this is another type of prompt
which is the exact same thing as the experts prompt,
but with just regular people,
so Personas Prompt is what I call it.
Name a number of demographic personas
who would be relevant for answering this.
What we're trying to do is get like a crowd of people,
and instead of them being experts,
we just want regular people.
And what are the relevant demographics?
So in this case, we do have a footwear industry expert,
but we also have a busy parent and an athlete
'cause they're gonna have different perspectives.
Then it's a fashion conscious person.
So we're gonna tackle this from multiple angles.
You can say it does sound, you know, direct and trendy,
but it doesn't specifically mention footwear.
Omnifit might be too vague, in this case.
I suggest convenience for busy parents,
however, it doesn't clarify the product's purpose.
Getting a much better broad sense
of the problems with this name,
which I think you didn't really get in the previous ones.
And then, in this case, it says we do think
it's a promising name,
however, it needs to clearly convey
the product's unique selling position.
I think that that's a much more powerful final response,
in my opinion.
Cool.
So bringing all that together,
like here are the three responses side by side,
and in general, whenever I run this,
even for any type of question,
I'm finding I'm getting much better results
with the persona one
and sometimes the expert one as well.
We can judge this, like we can use an LLM judge.
In this case, this is just another prompt
where I'm asking it to compare the two responses,
and I want to make sure that the responses are unique,
that they're non-obvious
and that they sound human, not like an AI.
And then it's gonna choose which one is better.
This is like a common format for judging responses
because I wanna run this across a bunch of questions
and not just one question,
and I don't wanna just like manually check each one.
So I'm automating the checking, in this case.
This code basically just runs,
it just runs the judge prompt
and we can see if we just run this now.
In the naive versus experts prompts,
I chose the experts, number two.
In naive versus personas, it chose number two.
And then in experts versus personas it chose number one.
So what that means is they were both an improvement
over the naive prompt,
the first one we tried,
but the experts one was better.
And it comes with some reasoning here.
You can see like why it chose that,
but it says response one provided
more comprehensive analysis, which is useful.
Cool.
You'll get something different when you run it
and especially if you change the question,
but I find the judge also is pretty good at agreeing
with what my assessment would be,
which in this case I did think
that the personas one was a little bit better,
but in this case, it's pretty clear
that they're both better than the naive prompt.
All right, so bring those together.
I'm just gonna run this.
All this does is take all the things we just did
and put it into a function called run_test
that takes in a question.
And we're just gonna run through every question here.
So there's five different questions.
And we just wanna see generally how often does
the persona prompt win versus the expert prompt.
And so, right now we're just setting up the naive response,
and then setting up the expert's response,
and then setting up the personas response
of that same question,
and then it judges them over here.
This is where it's judging.
I mean it's printing stuff out for us,
and then it returns to final results.
So you can see that, in this case, we have the question,
what's the best way to learn a new skill?
And the naive response is useful,
but the expert response,
it pulled in all these heavyweights.
All right.
Pulled in Malcolm Gladwell, Angela Duckworth,
Ericsson K. Anders, Tim Ferriss.
That's the first, it's pretty common.
Actually, all these questions,
but also people like Rich,
and they all have a different opinion.
Malcolm Gladwell talks his shop.
You know, he know talks about the 10,000 hours.
Angela Duckworth talks about grit,
and her book was called "Grit."
And so yeah, as you expect,
but because you're forcing it to be a little bit broader,
it does tend to work a lot better.
You can see, at the end here,
the actual personas response is quite useful.
Actually, we can pause, open a text editor,
and we can see here goes the final response for personas.
And yeah, I think this is really useful.
Like it talks about structural learning,
hands on practice, community engagement.
This is the sort of thing I would recommend
people do as well.
And practical skills, diving into real world projects
and seeking mentorship.
That's exactly what I do to learn a new skill.
And you can see as well that, in this case,
the personas prompt beat everything.
The experts beat naive, and then personas beat experts.
Let's go on through the rest of the questions now.
We'll see how it gets at the end.
What we're gonna do at the end is
just add up how many wins each one got
and then also how many times they would test it
when we run naive_vs_experts.
And then we'll count one for naive, in terms of trials,
and one for experts, in terms of trials.
And then we're also gonna count, in this case,
if naive won,
and then we're gonna increment the wins for naive,
or if experts won, then we're gonna increment the wins.
So what will happen is we'll get the accurate percentage
of how many times did it
when it was being tested against one of the other prompts.
And we're just gonna see what that looks like at the end.
Okay, everything's finished running now.
So I'm just gonna run this and here we go.
So on average, across the different trials,
the naive prompt only won one in 10 times.
So sometimes it is the better one, but it's pretty rare.
Experts prompt was much better.
60% of the times they won.
And the personas prompt won 80% of the times in the trials.
This is pretty common, actually,
I'm finding across different questions.
I try this on when I'm measuring it,
the personas prompt actually tends to be the best
and it's like wisdom of the crowds,
so I'm not sure.
I think with the experts prompt,
it does tend to be better than what you naively get,
but it sometimes goes a bit over the top
with the details of what Tim Ferriss,
or what Daniel Kahneman, or whatever.
All these famous people will specifically push
like their 10,000 hours,
and I think that harms the diversity of thoughts sometimes.
Whereas when it's looking for regular people
and it's having to be a bit more creative
with how those people would approach
answering that question.
Also this is just meant to be a way
to demonstrate how it works.
I use this technique in lots of different ways.
So just like chain of thought,
it doesn't necessarily have to be applied exactly like this.
It's just more of a concept that you can try.
And you can do this in ChatGPT as well.
But in general, what I tend to do is
I'll create the personas first as like a separate step,
and then I will run each one of those in parallel,
and then I'll aggregate at the end.
So that tends to be how I approach it.
I just wanted to show a simple version here
where it just does it all in more prompt.
Cool.
Yeah, hopefully that's useful for you guys.