-: We're gonna go a little bit advanced with SAMMO.
We're going to be talking about minibatching and metaprompts
and this is really the way
that people recommend using SAMMO.
And we're gonna pull in here the OpenAIChat,
which we need,
and we're gonna pull in the
DataTable
and we're going to get,
also need
an EvaluationScore
and
os,
pandas,
pd.
Okay, we're going to set up our runner, and this is,
you know, there's no difference to how we do this.
This is typical for SAMMO,
but we're also gonna define a function which loads the data,
because we're gonna pull our data from a spreadsheet.
This is, more likely, where you're gonna have your data.
In my experience, this is what I usually have,
some sort of CSV that've gotten from team
with all the inputs or outputs, examples,
and then you're gonna pull that data back
into a DataTable, and that puts it into the SAMMO format.
I'm gonna say, oh, sorry, it's from_pandas,
and then there's a few different things we need to do,
is that one is, we need to add the input_fields,
in this case, it's the "description".
We're going to pull in...
Actually, this doesn't need to be
a list,
can be a string, 'cause in this case we just have one,
and then we're gonna put in the "classification",
and then
the other thing we're gonna add is the constants.
So, and this is like stuff
that you can pull into your prompt afterwards,
that will always be the same
for every row in your DataTable.
So, in this case, we're gonna use it
to put our "instructions" for our prompt,
'Determine how to classify these transactions',
Okay, and this is gonna make more sense in a minute.
Let me see, return mydata,
so, that's how we're gonna get our data,
and then we just need one more function,
which we're gonna use, and then we'll dive into it.
So, we need an evaluation metric,
which we're gonna call accuracy
and we take in two different things.
We're gonna take in y_true, which is gonna be a DataTable,
then y_pred is gonna be a DataTable,
and then we're gonna pass out an EvaluationScore at the end.
And the way that will work,
is we're gonna say y_true = y_true.outputs.values
and then same for y_pred.
So, we're just gonna get the values
and then we're gonna do this EvaluationScore,
well actually, we need to say how many of them are correct.
So, we gonna say, and in this case it'll be calculate,
and that would be n_correct, let's call it that.
So, we need to sum up all the situations where
y_p == y_t for y_p,
y_t
in a zip of the two things.
So, if that doesn't make any sense,
I'll just explain it again.
We're gonna pull in the y_pred and y_true,
then we're going zip that together, so they're in tuples,
and then we're gonna grab the values of those tuples,
y_p and y_t and we're just gonna check if they're the same.
So, simple, kind of output here.
And then we're just gonna return the EvaluationScore,
which is just
n_correct / len(y_true),
so that's how many are correct.
Okay,
great stuff.
So, these are the two functions.
That's what we need,
and we've got everything we need now to do this.
load_data, let's see what the data is.
So you can see here, this is a bunch of transactions,
cash deposit, withdrew money,
insurance claim refund, whatever,
and then some classifications.
So this is what we want to call those transactions.
We also have our instruction from the DataTable,
so we're gonna use those prompt.
All right, so how do we send up our prompts?
So, this is how to do it manually.
This is the standard way.
Then I'm gonna show you the metaprompt way,
which I think is much more powerful and easier to read.
Don't worry if you find this a little bit hard to follow.
We're gonna import Output
and then we're gonna put the GenerateText.
We're gonna get the extractor, the ExtractRegex.
That's what we're going to use for this.
And then we're gonna import Template,
and we need to set up our prompt,
prompt,
and that's gonna just be the GenerateText thing,
and it's gonna be a template inside,
and I'm just gonna copy this template across.
So, what do we have in our template?
We have two things.
We have...
We have the instructions.
We're gonna get the constants.instructions
that comes from the DataTable,
and then we've told it what output labels we want.
These are the different classifications it could be.
It could be Rent, Other, Food, Entertainment, Utilities.
And then we've just added in this inputs here.
This is basically iterating for each input.
It's gonna create a prompt,
and then it's gonna take that value and just place it.
We're gonna say input and then the value.
So, you know, if we iterate through here.
It's gonna have like, cash deposit at local branch,
withdrew money for rent payment,
withdrew cash for weekend expenses, pull all those through
and it's gonna iterate through them with this each.
This is like a foreach function,
but the cool thing it's gonna do is,
'cause we set it up in this way,
we can do a batch of them at the same time.
We're not just doing one at a time.
It's actually doing, you know, say 5 or 10 at a time.
Let's just get our data.
We're just gonna take a sample of that data,
10 different
and seed here, so you get the same response hopefully,
and then we're gonna say, set the labeling_output
and to do that, we just need to ExtractRegex.
So we're gonna pass in the labelling_prompt.
and then we need a RegEx, which we'll grab,
with the different things that we've identified.
So, in this case,
it should be this.
And that's the RegEx
that can pull out the different responses here.
And then, I'm gonna say
minibatch_size=10.
So, it's gonna do one batch of 10 all at the same time,
and put that into result,
and result looks like that.
I put this in the wrong place,
put it in the RegEx, it should be there.
Here we go.
So we've output here, the actual response from that batch
and we've only run one batch here.
So if we said we're gonna do a minibatch_size of one
that will do 10 API calls
and it's gonna run them all at the same time,
or if we wanted to do all 10 at the same time,
we can do that.
This is a really powerful way
of, basically, iterating through a lot of data
in a quick response.
So, that's minibatching
and that's a really powerful way to do it, right?
And then we can also figure out the accuracy as well,
See if it's...
So let's check, see the sample result.
Yeah, it's a 100% accuracy,
because we're passing the sample, passing the result,
so that's really good.
But we can scale this up as well if we wanted to,
now that we have this program.
We could increase the sample.
We could change the minibatch_size.
Now, let's look at the prompt itself.
So, we wanna just call the plot trace on this.
We can see what's going on.
Right, here We've got the instructions.
They're coming into that prompt.
We already had the output labels and the prompt
and then we have the input, cash withdrawal for a holiday,
mortgage payment, ATM withdrawal, cash.
So we've passed in all of these things in the minibatch
and then it's given the output here,
and it's come out in a non-standard structure,
but that's fine 'cause our RegEx could handle that,
and it's pulled through all these different answers,
which is really great, and that's how it works.
Alright, so that is how you would do it manually,
but there's a much quicker and easier way to do this
through metaprompting, which I find much more useful.
This is essentially the same thing.
I'm just gonna copy and paste this across,
but there's a few different things that are different.
One is, we're setting up this MetaPrompt structure.
We get that in from the instructions.
Great.
So, from sammo.instructions we're getting MetaPrompt,
Section, Paragraph, InputData, FewshotExamples.
There's a bunch of different helper functions.
There's a few things that are interesting.
One is, we have this instructions section,
and we're just getting that from the constants,
so that's not different,
but here we're adding in some FewshotExamples
just to show you how quickly you can do this.
It actually just pulls from the sample, from the data,
and it pulls in these examples, which is really helpful
because it's already formatted in the right way
for that dataframe.
We're also adding Paragraph to show the Output labels
and then making a placeholder for the InputData.
We can render it as "markdown"
and then we're adding a data_formatter,
which is gonna do the RegEx for us.
We don't have to know how to use RegEx, which is nice.
We just put in our list of the different things it could be,
and then we can also say what happens if there's a failure.
We should get an "empty_result".
So, what does this look like?
If we just look at the program,
it's, basically, exactly the same,
but it's just pulling all these different things together
into a MetaPrompt,
and it has the FewshotExamples as well,
which we're gonna see in a second.
Alright, so let's run it and just make sure we get the same.
To run it, we're gonna say result,
and I'm gonna paste and put in...
Let's just do five this time.
Let's say, on_error="empty_result".
Let's say run(runner, sample), not the full data.
And then, so that's done,
we're just gonna check what's the result look like.
Here we go.
So, it worked exactly the same.
In this case it did two batches,
'cause we said minibatch_size=5,
and here's just an example of how it worked.
So, we have exactly the same prompt going through,
except we added these FewshotExamples,
but it's a much cleaner way of looking at it, right?
Compare that to the way this is.
It's very complicated to me.
I'm like, it's got lots of variables in there,
and it's like pretty hard for me to parse,
whereas the nice thing about this is,
I'm like, I can edit this so easy, right?
It's just a list of different sections.
I got the Instructions Section, I got the Examples,
I got the Output labels,
and then this is where the input stuff's gonna go.
So, that's much nicer and easier in my experience,
and this is how I tend to use samples
to string together these examples.
And then that way, if you're like,
"Oh, I don't want the the FewShotExamples."
Let's see how that works, now.
It's just gonna be the same,
but it's gonna be a slightly different prompt rate,
and then, we're gonna... an error in this case (laughs),
but I put those back in.
Now it's working.
But you can quickly swap different instructions in.
You could say something like,
"Make sure you use the right labels to the transactions."
So you quickly change the prompt,
and then it's gonna run again,
and you can see if it's done the right thing.
So, that's the benefit of this.
You can just swap these sections if you need to.
And then one more thing, which is quite helpful,
you may not realize,
and I find this is also useful for observability.
In DSPy, I find it quite hard to see what was actually up,
what where the LLM calls, different requests and things
that were made,
but it's much easier to access them with this.
Here, I just put into the result,
and then I've got the, you know, llm_requests.
So, if you just go into that list you can see,
okay, here's what's actually it's compiled into.
So, we can see we've got the examples,
see the FewshotExamples that that was in,
got the output labels,
and then these are the inputs
that we want it to do something with.
So, I find that's like much cleaner to be able to see.
The other thing we can do, which is quite nice,
is we can change the prompt relatively easy,
modified_mprompt in this case.
We just take the nprompt and we clone it,
and then we have to use this function called rebind,
but what that's gonna do is take the exact same prompt
and then just change one aspect of it.
So we're gonna copy in, change this to "data_formatter",
to a JSON data format,
and then we'll run it exactly the same.
We can see that the results are the same,
but when you actually look at the compiled prompt,
it's JSON.
It gave the JSON examples instead
and it's passing in the JSON.
This is really nice to swap things in and out.
You could also swap models in and out as well,
which is really helpful.
One more thing I wanted to show you here
is that you can actually do RAG as well.
So they have support for local RAG.
I haven't used in production. Take it with a grain of salt,
but for a lot of experiments
and things you're running locally,
I think it's quite helpful.
So, you just need to make a embedder,
and then let's called OpenAIEmbedding model,
and then you just pass in,
this is just like creating a runner,
pass the cache file, etc.
I'm just gonna set a rate_limit.
The rate_limit's gonna be 10.
That's all I need, I think, for the embedder.
Actually, let's call this
"embeddings.tsv"
and let's call this "EMBEDDING_FILE".
So we do some conflict prevention, cool.
Now it's gonna do, you know, the same thing.
It will get like the embeddings.
Instead of calling the OpenAI generative model,
it's gonna call the text embedding model.
It's gonna get the embeddings for your different things.
Let's look at what data we have,
just gonna find what's the len(mydata),
and split it into fewshot and training.
Here, we just split it into training data,
which we're gonna use to kind of test against,
and then fewshot data, which is the examples
that we can pull into the prompt dynamically.
Before we had these FewshotExamples,
but we're not sure whether we're pulling the right ones in.
Well, we then can actually do that.
We can make that happen with embeddings.
So, the way that we sort our data there,
just if we wanna see what that data looks like.
So, let's just slice that dataframe,
But then we're gonna pull in the
EmbeddingFewShotExamples,
sorry, FewShotExamples,
and instead of you doing the FewShotExamples
that we had before, we're gonna swap that out.
If we just grab our prompt from before.
This is our MetaPrompt.
So, we can just get that,
but then instead of this FewShotExamples,
instead we're gonna call that EmbeddingFewShotExamples,
and what do we need to pass in here?
We need to pass in the embedder first,
and then the fewshot data,
and then the examples.
and then
budget="relative",
and with these parameters what it's gonna do
is it's gonna to take the embedder,
it's gonna find all the embeddings
for everything on the dataset,
and then it's gonna pull the three most relevant examples
for these inputs.
So, chances are you actually get the same transaction,
that's already been classified, which is really helpful.
So, that's what this does.
And then we're just gonna call this,
instead of mprompt_parsed,
we're gonna call this mprompt_rag_parsed,
and rag as well, just to separate those two things.
and if this works then we're gonna see
that it's pulling in different examples here.
So, that's run in exactly the same as we did before,
but it's got,
plot the thing,
the EmbeddingFewShotExamples.
You can see it's pulled in insurance claim refund, Other,
bought coffee and snacks from cafe, food,
pay for streaming service, entertainment.
So, it's pulled those examples in, and that's good,
because the input examples are insurance claim refund,
bought coffee and snacks.
So, it actually found a lot of the same transactions
from the RAG database.
So, the smart thing here is,
that as you classify things correctly,
it's gonna build a bigger RAG database,
that you can pull from.
So, the chances are,
if you have a lot of corrected examples,
you're pulling in these.
Your FewShotExamples will match the inputs perfectly,
which is really great,
and it means that it's very unlikely to do something.
So, that's a really powerful thing.
I haven't explored it a lot,
but to be able to just do RAG locally like that,
without having to mess around with Face, or Chroma,
or whatever, is really cool.