-: Hey, welcome back.
And so basically in this video we're gonna have a really
long Jupyter Notebook talking about how you can build
customer valuation metrics for LMs.
We're gonna explore a different couple of options
that you've got and we'll look
at some of the trade-offs as well.
And there's a lot more text that you can read
inside this Jupyter Notebook.
So the main focus is gonna be just
what are those different techniques
and also maybe some of the trade-offs
and showing you different implementations of those.
So let's start by talking about what are the different ways
that we can evaluate an LLM's response.
The first one is called, we can call it something like
programmatic rule-based evaluation.
And basically what you're doing there is creating custom
rules that determine whether or not that result is good.
So if you're thinking about the context
or a task of a blog post,
that could be creating an e-bar metric like does it contain
more than a hundred words or is it a certain length?
And we can obviously create, some of the advantages for
that is it is a quick feedback loop and it's cost effective.
The disadvantages are that if the task is very nuanced,
then you're maybe not able to actually evaluate the quality
of the LLM output using such a rudimentary metric.
But the second one that's quite interesting is
LLM based evaluation.
And so what you're really trying
to do here is say "If this task is too nuanced
to create a Python function that would evaluate,
or a series of Python functions
that would evaluate performance."
And so we can use another LLM
after we've produced a result to then evaluate whether
that thing was a good result or not.
And so this is really helpful when you are trying
to do things such as evaluate helpfulness or readability.
Those metrics are too vague and too nuanced on language.
So we can almost ask an LLM to generate 10 examples
or 10 outputs, and then we can ask another LLM on top of
that to then rate each of those outputs.
And the third way is basically doing human-based evaluation.
Now, human-based evaluation is basically
where you produce a load of outputs from your LLM
and then you'll ask a human to rate those.
And there's two different approaches to that.
So you can either do like a simple approval
and disapproval system,
or you could say to the human, "Which of these three
do you do you like the most?"
And obviously with the human based evals
you get the highest amount of accuracy.
So it's really good for tasks that have a high customer risk
and getting false positives
or false negatives is gonna cause consumer damage.
But the disadvantage is obviously there's a labeling step.
So feedback is a bit slower
and also it's time consuming to do that
and there's a cost associated with that.
And so it's really good for best to high stake tasks.
And so that's like a brief overview into those three
different types of methodologies that you might decide
to use when you are creating these custom eval metrics
for your LLMs.
Now I wanna just go through
and so we will start by, there's also a little bit
of text here if you want to talk about the fact
that you can start off with one eval strategy
and then move on to another one.
And you can also minimize human error by
specifically on the last one when we're talking about doing
a human labeling, maybe you get three evaluators
to rate the examples rather than just
relying on one labeling, a person.
So there's a couple of different tricks
and tips you can do there.
So let's look at programmatic rule-based evaluation.
So in this, what you do is we've got you setting up Pandas,
we're using NumPy and this TQDM package.
Now we've got some LLM results.
I just thought I'd hard code some of these.
And what we're trying to do here is produce
a Twitter or an X post.
And so what we wanna do is we wanna
evaluate these results from the LLM and we wanna see whether
or not these are useful or not.
Now one thing that is obvious is,
and maybe we wanna see whether the post has a hashtag
and we also might want a suitable length.
Like if the post is, the X post is too small,
it's probably not a good use of,
that's a good post.
So we'll first just load those into a Panda data frame
and then we've got the social post DF data frame.
So you can have a look there and you've got the text in this
generated social media post column.
Now the first thing we're gonna do is we're gonna set up
our two eval metrics.
And just so I was saying earlier like let's say
that we'll define a good social media post
that they must contain at least one hashtag
and they must be greater than or equal to 30 characters,
but not more than 200 characters.
So we can set up these two individual eval functions
that takes in the LLM text that is produced.
And then we're gonna run that through a series of steps.
Now I've also got a nice helper function here
that's just gonna take a Boolean result in.
And if the Boolean result is true,
then we're gonna return one, else we're gonna return zero
and that'll be quite nice for doing some counting so
that when we come to calculating the accuracy scores.
But we'll run those functions
and then we're gonna set up these eval functions inside
of a Python dictionary.
And then we're just gonna loop through all of the rows.
And for each individual eval function,
we're gonna figure out what eval function we need to call.
And then we're gonna just pass in that text,
get the eval result.
So this is basically saying like for each individual row
of the social media text, let's run a function on the text,
get the result of that which will be a Boolean value,
and then we're gonna pass that into our helper function,
which is gonna take that Boolean result
and basically update that specific column in
that specific cell with either zero or one.
So let's go and have a look at this.
So you can now see we've ran that on
and we've got the first five rows here
using the dot head method.
And you can see like we've got this eval has hashtag column,
so you can see like the first result doesn't have a hashtag,
but we know that these ones actually do have hashtags.
And similarly we've also evaluated on the length
of a social media post.
And so all of them in the first five results
are all an adequate length,
but we've failed on that first one.
So the great thing about these individual eval functions
is we don't have a cost associated with these.
They're very similar to unit tests in the sense
of we can run them very quickly,
we've got a fast feedback loop,
which is really, really good.
So let's move on and see how can you calculate the accuracy
for each evaluation column within the Pandas data frame.
So we're just gonna get, go do a list comprehension
and just check for this eval,
which will just give us all the eval columns
and we can just have a look at that.
So that's just getting all the eval columns.
And then we're gonna set up a Python dictionary
called eval accuracy results.
We'll loop through every eval column
and then we'll just get that.
And then what we're gonna do is we're gonna sum up
the number of ones in there against the length
of that individual column.
And then we're gonna just save the result of that
to a Python dictionary and then we can print out that.
So in our case, we've got around about 77%
eval accuracy on the social media posts having a hashtag.
And we've got around 44% of the eval length form,
unlike the actual social posts itself.
Different accuracy ratings based on different eval metrics.
Now just before we move on to the other strategies,
I think it's worthwhile talking about the fact
that we might wanna take a blend
of these two different types of eval metrics.
So we might wanna look at the mean,
or we might want to say,
"Let's actually weight these differently.
I care more about one metric than the other one."
So let's just go and have a look at how we could do that.
So in this scenario, we've just got our Python dictionary,
we're just gonna take the NumPy dot mean
of these two values and that gives us a mean accuracy of
around 61% accuracy rate.
But we could also actually decide,
I really care more about my LLM outputs having a hashtag.
And so what we could then do is we could say,
"Well I'm gonna say that the weight of that one
should be 70% and the weight
of the other one should be 30%,"
and then I can just get a weighted average accuracy.
So my accuracy's gone up to around 68%
and you could play around with these weights depending upon
what's more important for you.
So there's a couple of different approaches there
and obviously you could have multiple eval metrics.
So let's move on to the secondary strategy,
which is using another LLM.
After you've got some results from your LLM,
you're gonna use another LLM to figure out whether
or not the thing is valid
and then you're gonna compare that against your true labels.
The first thing that I just wanted to say is,
in this example
we're gonna be comparing against whether
those previously social media posts are a topic of coding
or software engineering.
And so the first thing that we need to do is we need
to add on the social media labels, like the true labels for,
is that specific social media post
actually talking about software engineering or coding?
That specific topic.
And you'll see, so what we do here is if I then print out
this social post df,
what you'll see is when I get scrolled back up,
so like for example, when I say "I love my dogs,
I love my dog," that's definitely not
talking about software, right?
But like all the other ones we've sort of said "Yes, give
that a 1 because it's definitely
talking about software," right?
So we've got a couple of posts here
that like really aren't talking about software
and we've just added the true labels to those
because we still need to reference whether
or not the evaluation metric was right.
And so we're still generating some true labels
and at the moment we're doing that manually.
And now that we've got the true labels,
we can then ask ChatGPT.
So we're gonna ask ChatGPT 4.
So we set up a chat model,
we'll set up the prompt template
and notice we're passing in things
like the format instructions and we're talking about
what the generated social media post is gonna be.
So that's the post to analyze.
And we've also got like the topic
that we want to classify against.
And so after we've set up that prompt template,
the next bit of the puzzle
of making a social media post classifier.
And so we've got, this "is" topic, which is an integer
and it defaults to zero.
And we've put this field is a classification result,
whether result is identified against a known topic,
one for yes and zero for no.
So telling the LLM,
give us a 1, if it is gonna be the coding topic
and give us the 0, for it's not the coding topic.
And then we then basically create that output parser
and we create a LangChain LCL chain.
And then we're gonna set up a results Python list
and we're gonna say this is the topic, right?
So coding, software or data science,
we loop through each individual row
and basically what we have is we're parsing the topic,
the output passes format instructions
and then also the social media posts.
And we're just gonna wait for that to run.
And obviously what you can see here is
that when we're using LLM based evaluation,
there's obviously a latency effect here
where you can't get those quick evaluation metrics
and also there's a cost as well.
So we then added on that.
So we can see, we call this eval is coding topic.
And you can see here for example, that it's very accurate
but it has actually made a mistake on this last one
where we know that it's not a coding topic
but that the GPT-4 has decided it is.
So what we do is we get the true labels in one variable
and the eval results in another
and we just create a count variable starting at zero
and then we loop through social post DF
and we just check like this row does it compare to this?
So we're just checking row by row
do these things, are they the same?
And if they are the same, then we increment the count
and then what we do is we take the count
and we divide that by the length of the true labels
and then we can see, so GPT-4
is getting a rough accuracy about 89%
against the true labels.
But the disadvantage
of this strategy is obviously we still had
to add the labels manually.
Now that can be quite tedious.
So we can take it a step further
and we can say maybe let's use GPT-4 to
synthetically generate the right answer.
And we could go in and change those synthetic answers later
and to catch various false positives or false negatives.
But we could use GPT-4 to create synthetic data.
And then after that we could then now use something like
GPT 3.5 to evaluate against
that grounded truth labeled data.
So I've got some transactions here
and we're just gonna load those in.
And then what we're gonna do is look at these.
So you've got like a cash deposit, a local branch,
you've got like "withdrew money for rent payment"
and we've got like a "withdrew cash"
and we've got various different types of pieces of text
that represent what happened at a bank transaction.
And so what we now do is we've got pretty much
the same kind of strategy.
We've got ChatGPT-4 and JSON mode
telling it that you can analyze bank transactions
and here's the transaction to analyze.
We set up our prompt template and also as well as
that then we've got a different pedantic model,
which is getting the transaction type
and also getting the transaction category.
Now we pass that to the output parser
and we set up our LCL chain
and then again, we're creating the results
and then looping through the data frame,
we're then appending the results.
So I'm just gonna leave this to run now
and that's gonna take a bit longer 'cause it's using GPT-4.
But the important point here is that you want
to use a very powerful model
to create these ground truth labels
'cause you don't want to create labels
that aren't actually correct.
And the other trick here is that when we come to evaluating
against these ground truth labels, we're going
to use a a cheaper model, a less performant model
because that's not gonna get 100% of it right.
And that allows us to then work on optimizing the prompt
and sort of making sure that we can really get,
if we can get a smaller model working well
through optimizing the prompt
or optimizing the retrieval, then
that means we can take all those learnings
and put it back into a more powerful model.
So in this scenario we've got two things.
We wanna store the transaction type
and the transaction categories.
So we're gonna just loop through the results and store that.
And so now we've got our grounded truth labels
on our transactions.
Now what we now need to do is we're gonna go
and test the accuracy of GPT 3.5 turbo against that.
So it's pretty much exactly the same
but we're just switching out the model here.
And again, we're using this GPT 3.5 turbo model
and creating our chain.
And then we're just basically, yeah,
just getting the transaction subscription
and then if there is an error from the parsing,
where we're just basically saying
the transaction type should be null
so we didn't get that one.
And then we do the exact same thing here.
So getting those results
and then basically printing those out.
So let's leave that to run.
And again, you can see there's a slight latency
involved in this,
so that will take about maybe 10, 15 seconds.
So we'll just wait for that.
So whilst that's running, we're saving the data
to this GPT 3.5 transaction type category
and also the the GPT 3.5 transaction category.
And we are gonna compare those two columns against the
grounded columns, the grounded truth labeled columns
that GPT-4 created.
Now let's go
and have a look at the code whilst this is finishing up.
So we've got a transaction type counter
and a transaction category counter.
And so what we're gonna loop through each row
of the data frame and we're gonna see, okay,
does GP 3.5 transaction type,
you pulled the transaction type row
and the same for the transaction category,
and we're gonna increment those counters
and then we can work out what the accuracy is.
So GPT 3.5 is giving us a transaction type accuracy
of about 0.9% against synthetically created data from GPT-4.
And we're getting around about 75% of accuracy against
the category accuracy.
Now let's go and have a look at the third type,
which is human-based evaluation.
So obviously that's where you would get
some immediate feedback from your team,
and they would do the labeling manually
or you would use a cloud vendor solution
like Amazon have got a Amazon Mechanical Turk
and they've got a data labeling service and GPT
and Azure have also got data labeling services.
So if you're not
specifically interested in waiting a long time,
then you could pave RN API to get a lot
of the data labeling done for you.
And that would help speed up the feedback loop.
But let's dive in and have a look.
A good example of maybe we're generating some images
and it's for a mortgage offer
and we want to personalize this
and we wanna make sure that each of the images is good.
So what we're gonna do is just use the open AI package
and we're gonna generate images using Dolly 3
and then we're gonna set up our standard chat model
and then we're gonna tell it, "I want you to generate
a really good image prompt using some customer information
and customer profile."
And we've got a couple of different things that we want
to make sure that the visual prompt is concise,
avoid including any sensitive information
in the prompt, et cetera.
So we're really trying to get some good housing images
for our mortgage deal for the bank.
So what we now have
as well is we've got a Python function here
that's gonna take in that visual prompt
that it generates and it's gonna generate an image
using Dolly.
And it's also gonna save it to this out directory as well.
So that's gonna generate the image using Dolly 3.
And then we're gonna get the JSON version of that,
the basic 64 JSON
and we're gonna download that image
and save that and decode the basic 64 image
and return the image path.
And so we set up our image chain.
This one is taking a customer information
and a customer profile.
So we need to parse both of those
when we invoke this LangChain LCL chain.
And that gets past the image prompt,
which then creates the image,
which then we have a prompt which gets piped into the string
and then that string, so that visual prompt
then gets injected into that Python function.
And then from there then
what you can do is generate an image.
Cool.
So we've got that set up
and just to save some time, I've created some good profiles.
So these are some examples
of different things that we'd expect to see
and the outputted images.
And you've got like the profile
and the information about that.
And so we've got a Python list of some customers
and various things that we want to see.
So we're just gonna loop through and generate some of these
and then we're gonna store those.
So you'll see a couple of different things going on here
where we've got that image chain that we're invoking
and we're parsing in the customer information.
So that's that profile information and the profile.
So that's just coming from each
individual Python dictionary.
You've got the profile key here, the information, notice
how we're lining it up with what the
LangChain prompt template expects, right?
So if I scroll back up, whilst that's running, you'll see
that it expects a customer information
and a customer profile input into that prompt.
Cool.
So that's now running, it's gonna take about a minute
to run 'cause we're doing just a couple of customers.
We're also saving the image path,
which is the output from this final chain.
We can see that that's the output by, if I scroll up here,
this generate image,
the final thing it returns is an image path
and that'll be inside the out folder, right?
So then what I'm gonna do is I'm gonna scroll back down
and this will finish up in a second,
but now that we've got those images,
what we wanna do is we want to manually evaluate that
and that's where the human will start coming into the loop.
So they'll actually generate a load of results
and it might not necessarily be images,
it could be text, it could be video content.
Your evaluation is not necessarily tied
to a specific content format,
but in this example we've done images.
But this same process would work
the exact same way with text you generate,
your LLM results or your AI results using generative AI.
And then what we do is we've got like a nice interactive
approval system, a couple of different functions here,
like on the button clicked, it's gonna run some code,
and it's also gonna update the response as well.
And that will be updating the data frame.
Feel free to take a look at that code if you want,
but you don't really have to fully understand it.
But just know that it's gonna generate some widgets
inside of a Jupyter Notebook.
But let's actually run it and see.
So you can see the image each time,
and I like that image, so I'm gonna give that a thumbs up.
I think this one's may be a bit too corporate
and I think the text here is hallucinating.
So I'm gonna give that a thumbs down, what it could be,
what it could, new mortgage deal.
And it's spelt mortgage wrong here,
so I'm gonna give it a thumbs down.
And I also don't like this one.
And then I quite like this one.
I think it's good, it's very playful
and there's no text in this.
So I'm gonna give this a thumbs up.
And now you can see here we've got
the number of approved images.
So we've got about two divided by five accuracy.
So you've got around, let's have a look at that.
So it'll be around 40% accuracy out of our images
and we could keep changing the prompts
and changing the different image models to try
and improve the accuracy.
Now I just want to talk as well about the different ways
that you want to think about,
now that you've got these evaluation metrics,
how you could specifically optimize using any
of those three strategies.
So the first one is obviously optimizing the prompts, right?
Can you add extra information
or using the prompt engineering principles
or adding extra context
to the prompt using things like retrieval from vector
databases or Postgres or MongoDB.
So thinking about what can you include into the prompt
to make the prompts more effective.
Now the other thing you could play around with
is the second thing, which is model selection.
So you could use a more powerful model if you're not getting
the level of accuracy or the high level of accuracy
that you're looking for.
The third thing we've already covered are using more
advanced or enhanced retrieval techniques,
which will then feed into the prompt.
So that one's kind of related to number two.
And then number four is using task decomposition.
So rather than actually going
and deciding to put everything inside of one prompt,
maybe a better way is to break down
a very large problem into a series of sub-problems so
that you can then solve that more elegantly
with very fine-tuned prompts.
A very optimized prompts for that specific sub-task.
And the final one is using fine tuning.
So all of these different ways,
these three strategies will provide you with the ability
to run fine tuning.
And you could then use an open AI fine tuning job,
or you could tune, fine tune an open source model
by running that inside of AWS
or Azure or Google cloud platform.
So there's various different ways
that you can apply fine tuning,
but you could then use your own data
and your own labeling process or your own evaluation metrics
to then optimize and fine tune an LLM.
So hopefully this helps, we tried
and make a more of a comprehensive guide this time,
I'll see you in the next lesson.