-: Hi, let's learn about parallelization.
Let's install OpenAI.
Just gonna get the OpenAI client.
That's the main thing we need.
So let me just paste this in and we're gonna run this.
It's gonna ask you for the secret key
and you're gonna go into your account into API keys
and you're gonna get an API key and paste in here.
So if you don't know where that is, then here is the link,
platform.openai, api-keys.
All right, so now we have the secret key locally.
Now we can run our...
So just going to create the clients.
We're gonna create two clients.
One is the client and one is the async_client,
and I'll explain what that means.
But broadly speaking, what we're gonna show you
is how to do requests a normal way to OpenAI
and how to do it asynchronously.
Asynchronously just means it can run in parallel.
That means you could run one, two, three, maybe 100
or 1,000 API calls all at the same time.
And it's just gonna be a lot faster
than if you run them one after another.
It might not be obvious to you
if you didn't study computer science
why that might be the case.
But I think about it like a checkout line
in the supermarket.
If you just have one teller,
one person checking the groceries
before you go out the door,
like, you know, if you have one person
you can pay for the groceries,
then every single person in the store
has to wait up in a line for that person to finish,
and they're processed one by one.
If you hire more people in the checkout,
and if you have 10 of those people
where you can pay for your groceries,
then they can process 10 people at a time.
So even if one person's being slow,
like, all the rest of the people don't have to wait.
And that's essentially what asynchronous API calls is for.
And what we're gonna do,
I'm just gonna paste in a prompt here
from a project that I run.
So I run a company called Rally, askrally.com,
and we did this experiment with media diets.
And the way this experiment worked
is we tried to predict the US election
to see who voted for Trump,
who voted for Kamala Harris using personas.
So here we have Sophia Martinez,
she's a 40-year-old emergency room nurse from San Francisco.
I want to see who she would vote for,
Kamala Harris or Donald Trump.
And so that's the simple prompt.
You know, our actual system prompt
is a little bit more complexness, (chuckles)
but it's like a basically decent version.
So what we're gonna do is we're going to get the response
and we're gonna do chat.completions.create.
We're gonna pass in the model, which is GPT-4.1
and we need to pass in that system prompt.
So we're gonna put in...
Oh sorry, I already have the content, so...
Yeah, yeah, there we go.
It's gonna be role system and then role user.
The system prompt is going here
'cause that's telling it how to act as the persona.
And then you have the user prompt,
which is the question that we're asking.
The prompt format we're gonna use here...
Let's just get that back as JSON
and then gonna print the responses.
We also want to time it as well.
Let me just do the start time
and that should all run.
So we're gonna start the timer.
We're gonna get this response
and then we're gonna print out at the end.
How long it took?
Okay, so we've got an issue here.
Ah yeah, we're passing on the wrong structure.
Okay, here we go.
So they're thinking about the vote.
"Oh boy, it's not even a hard decision for me.
Donald Trump brings back all those chaotic nights
at the hospital mask fights, misinformation,
the tension between coworkers."
So not a big Donald Trump fan (chuckles)
as she votes for Harris, right?
So that vote came in two seconds,
which is really great and it's quite fast.
But if you needed to poll maybe 100
or 1,000 of these personas,
it's gonna take a long time.
One of the things we've done with Rally is we've set it up
so that it pulls lots of different personas all at once
and I'll just kinda show you how that works.
So what we're gonna do is gonna run_multiple_queries.
We're gonna count the total time taken
and we're gonna tally the votes, okay?
For i in range of number of runs,
so here, we wanna run 10 times.
We're going to measure the start time.
We're gonna get the response.
And then...
Yeah, we use the same json_object stop,
so that makes it easier to tally.
I'm gonna count the total time taken
and then we also want to pass the response,
so it's gonna count all the votes.
All right, and let's figure out the average time,
the results after a number of runs,
so we'll count the votes and then the total time taken.
Okay, so that's...
It seems well, so let's actually run the function.
And it's gonna run this 10 times.
It's gonna tally the votes.
So you can see it's taking more than two seconds now
'cause we're running it 10 times.
This is just to really show you how slow it can be
to run things in sequence.
And we're gonna show you async in a second.
So if it takes two seconds each,
it should take just over 20 seconds potentially.
But some API calls take longer than others.
So you can see here total time 26 seconds,
took 2.6 seconds on average
and it voted for Kamala nine times.
But if you're trying different personas,
you might get some votes for Trump.
It's also really useful just to see
how often this persona votes for Kamala or votes for Trump
because then you can average out the votes
and see if sometimes they vote for another candidate.
All right, so now how do we do async?
The way we do it is a little bit different.
We say async def instead of just def.
To create the formula, we have to make an async formula.
So we're just gonna call this make a single query,
measure the time, and then we're gonna get the response.
And it's exactly the same and always
except for just one small thing,
which is we have to await the response
and we have to use the async_client.
So that if we try and use the normal client,
that's not gonna work.
But in this case, you just need async_client
and you just need to await.
And if you use await within a function,
but you have this async at the beginning,
then that's gonna give you an error as well.
But that's really only major difference
for creating the function.
And then we're gonna return the vote and the time taken.
All right, so this is an async function
and that means you can run this multiple times in parallel.
So let's do that.
So let's just say run_multiple_queries.
And then the way that you do async
to make it run concurrently
is that you create the tasks.
So here we're making a single query for the range, right?
So this is gonna make 10 tasks.
And then you gather at the end with asyncio.
We just do a wait asyncio.gather and then *tasks.
And what that does is it spreads out the tasks,
it will run all 10 of them at the same time, so that's...
You can think of this as like setting the clock.
It's gonna start running and then it's gonna finish
and you'll get the results when all 10 have completed
or if one of them failed, whatever it is.
Cool, we're gonna count our votes.
We're gonna count the times as well
and then we're gonna iterate through them.
But yeah, it looks like it's predicting
the right things here.
Just looking at the code.
So it's gonna tell you the votes for us.
It's gonna tell you the individual times.
It's gonna give us the average individual time
and then let's just like print out some stuff here as well.
All right, so now we're gonna run this async.
Let's see if this works.
There we go. It was much faster.
It took four seconds.
What this means is it's on average gonna...
Like the whole 10 queries is gonna take
just as long as like the longest query.
'Cause they all start at the same time,
like starting a race, right?
They all start on the same startling line
and then when the last one crosses the finish line,
that's when we finish.
So sometimes the responses take a long time.
Like, on average it was 2.6 seconds this time
or on average it was 3.6 seconds.
But in this case, the longest one finished at 4.3.
So it was much, much faster than what we had before.
All right, where it took 26 seconds
to do the same number of queries.
There is a limit to how many you can run per minute.
It changes all the time,
so I'd check that in the documentation.
You could run hundreds or thousands of these.
You can see Kamala got 10 votes, we've got the same result,
but we just ran them in parallel.
Really useful concept to understand.
Definitely dig a bit deeper into that if you're interested.
But we have the ability now
to run lots and lots of things all at once
and that is very helpful for specific cases,
like what we have with Rally.
For example, I just ran a project
for, like, a custom project for a client
where we were running a survey for 5,000 AIs.
So 5,000 AI personas like this.
And you can imagine if we took two seconds each
for 5,000 personas,
that's gonna take us a couple of days.
Whereas, you know, I could run it in 10 minutes or something
because I was running it asynchronous
and I was doing a set of personas at a time.
I think I was running 100 at a time
and then that only let me run the whole thing
in just a few batches, which was really great.
All right, so hey, hopefully you find this useful
and you think about this
when you're running a long query like that.