-: Hey, I'm gonna walk you through
how to do qualitative analysis using an LLM,
but we're gonna use GPT-4o for this.
We're gonna use the API specifically
so we can do this without a lot of copy and pasting,
and so here's a script.
You can just run each line,
but I'm gonna walk you through it.
First you want to install OpenAI, and then if you run this,
it's gonna ask you for your API key.
You'll paste it in, and then you're ready to go.
If you don't know where to get your API key from,
if you go to platform.openai.com
and then you want to go into API keys here.
You can click Create a new secret key,
and then don't obviously share anyone. (laughing)
You know, it's like a password,
and people can spend money on your account
if they get access to it.
So I'm gonna read in this data,
and you can take a look at this data,
but it's basically just a bunch of interviews
from a paper that I found online,
and it was just interviews on like what young people think,
and so it's a perfect type of transcript to do,
you know, qualitative analysis on,
and all I've done really here,
just to be able to use this later,
is to create an id column, which is just the index plus 1.
So this is id 1, id 2, id 3, et cetera.
Gonna use that later.
Now if we run this,
what this does is just some formatting for our prompts.
So you can see there's 10 interviews here,
and I've just basically just stitched them together
and called them Sample 1, Sample 2,
Sample 3 using that id column,
and that's how the AI is gonna know which labels appear
but which ones are related to which document.
When it comes up with the new label,
it's going to give us the ids that label applies to
because we're giving the ids here.
All right, so here's the prompt.
I've kept it relatively simple.
I'm actually building a tool that does this.
So I've made this like a toy version,
but it's the same sort of structure in the backend.
I've just simplified it.
We're analyzing the given text samples
for thematic labels based on similarities and differences,
and I'm getting it back in JSONL.
JSONL is just JSON, but every line is a JSON object.
So this is what it's gonna come back with.
Okay, gonna run this just to see.
This is on the first 10 samples,
and we're gonna get back, here we go, some labels.
So there's a label called "Self-perception",
which seems to apply to everything,
and then there's one called "Anger/temper",
which only applies to sample 1, 3, 4 and 7, and so on.
Okay, then we're gonna be able to parse that stuff.
So here, I've just gone through each line, split out.
I stripped different things from it.
I got rid of the beginning line and the end line
because that was JSONL and then the back ticks here
'cause that would cause errors otherwise,
and now I've got like a list of these labels
that we can use.
Okay, cool. So that's how it works so far.
Now, obviously we want to be able to process
all of the things in the spreadsheet.
Sometimes they don't all fit into one prompt, so,
or you want to split them up a little bit.
So I made this little function.
It basically just shuffles the data frame
and then returns it in batches, and then for every batch,
it just creates a task with asyncio.
Asyncio is a way to run multiple API calls at the same time.
So we don't have to wait for a really long time
for OpenAI to come back.
We are not sending it and then coming back,
sending it, coming back.
Instead we're sending all the batches at once,
all those different API calls,
and then we get them all back at the end.
Okay, all the results gather here,
and then we can go through and parse them.
So here we go.
So here's a big list of all the labels we got back
from OpenAI, lots of interesting stuff,
but right now, we only got labels for that batch,
and we didn't get, each batch has different sets of labels,
and ideally what we wanna do is go through
and check if animal lovers applies to other batches as well,
check if cultural heritage applies to other batches
and dual identity, right?
So these are all the themes that are being discussed
in these interviews.
We wanna see which ones are more or less popular.
You can also see some errors here,
so like unique-to-sample-58, you know?
It doesn't always give you good results.
That's where prompt engineering comes in.
You can kinda mess around a little bit
and see what you can do to decrease that.
All right, and the next thing we need to do
is just modify the call_openai function.
We're just gonna add this labels=None,
and if we do have labels,
then we're just gonna add this extra piece to the prompt.
So it still works the same way for the previous stuff,
but we're just adding this piece here
that says you must only apply the following labels,
and then we just join the list together here.
There we go, labels_list,
and actually we're gonna put the labels_list in here.
Yeah, here we go.
So that's gonna give you the documents
with the label partial as well.
All right, I'm just gonna run that,
and then this is gonna go through every batch.
It's gonna create a task for every batch,
and then we're gonna gather all those tasks,
and then we're gonna parse the responses here.
We're gonna go through.
We're splitting the data frame again,
and then we're calling OpenAI,
this time with the for labels_list,
and then it's gonna give us back all those results
just like we did before.
So see if this works.
You know, this time, we've passed in the actual labels.
So we should get, you know, just those labels back,
but it should be for everything now.
So we'll get a big list of everything.
Okay, here we go. Yeah, there's a huge list now.
So we have like "self vs. perception" for this batch,
and then we'll have the same thing later on,
and so on and so forth.
So, you know, there's a big list of these,
and then this code basically
just merges everything together.
So what we get is, you can see here,
a big list of the labels for each of the rows,
and if I one hot encode the tags,
that kinda just pivots them up here.
Now we can see, like, we can filter in the spreadsheet
for self-description and then see, okay,
this one has self-description,
this one has self-description and this one does,
and we could just filter for how many we had.
If we look at the final shape,
we can see that we had 65 rows and we have 49 columns.
So quite a few of those if you, you know,
take off these six columns.
So yeah, 43 labels that we've found, which is pretty cool,
and you can export that to a CSV
and then upload that into Google Sheets
and do any other kind of analysis you want.
Okay, there's other stuff I didn't discuss here,
like you might want to de-duplicate these labels.
You might wanna check if they're any good or not or,
like, how interesting they are,
but this is the basic mechanism that I use
to do qualitative analysis with LLMs,
and it's pretty good at it.