-: All right, let's figure out
how to optimize our prompts using SAMMO,
so SAMMO is quite useful in terms of optimization.
In particular, prompt engineering
is what I find it's most useful for.
I still prefer DSPy, to be honest, for optimization.
I think it works a little bit better out of the box,
but I suspect that both of these libraries
will learn from each other
and steal each other's best features, I imagine, over time,
because there's a lot of similarities between the two.
Import SAMMO, and then we need to import, from the runners,
we need to get the OpenAIChat.
We also wanna get our components, so what do we need?
We just need output, I think, at this point.
We also wanna get the data table
and the evaluations,
and then import os
and import pandas as pd.
One thing I wanna do here, when we're doing optimization,
is just to set the logger as just "WARNING"
We don't wanna get a bunch of outputted text,
which can get a little bit annoying
but that's optional, you don't have to do that
so we're gonna set up our OpenAIChat
and it's a pretty standard setup
Here I'm gonna do just the timeout, 30 seconds
and then we're going to define a function for loading data
so the way we do that is we load the transaction data,
which is that file name in pandas
and you can get that from, get that into the format,
to define the input fields
Here it's just "description"
Output fields
which is just classification
If we had more than one, we would put this as a list,
but we don't need to
and then we just define the constants,
which is the instructions we're gonna use for all the data
and pull that into our prompt
and we just need to return mydata
Okay, we also need here,
I need to define a function for accuracy as well,
put y_true and y_prod.
These are both gonna be data tables
and then we're gonna return an evaluation score
so what that means is we're gonna get the y_true outputs
and then we're gonna get define n_correct,
which is just gonna get the sum
of whether y_p equals y_t
so y_prod equal,
and we're just gonna iterate through the zip
of the y_prod and true.
And so what this is basically gonna do is
it's gonna zip up these two lists of the results,
so they're both next to each other,
and then it's gonna iterate through them
and then just check whether they're the same.
And then, once you have that,
we can just get the final result,
which will sum n_correct
divided by len(y_true),
and that should, okay, the other thing we need
to pull in here, which I didn't pull in before,
we need to go from SAMMO instructions,
we wanna get MetaPrompt, Section,
Paragraph, InputData, and FewshotExamples
so that's gonna help us build our template
and we also need
from the dataformatters is import
the QuestionAnswerFormatter.
Okay, I think that's everything we need there.
Next do is we're gonna set up our labels,
which is just other food, entertainment, utilities,
we're gonna explain what this means in a second
but we're basically, this is like from a data set
that has transactions
so we need to classify into one of these buckets
so that's what our prompt is gonna do
so the set up a MetaPrompt,
and this is using basically just a list
of the different sections.
so the first section we're gonna have is instructions
and you can get that from the constant,
actually, yeah, it's, I've used this type of thing before.
We also got examples
The featured examples comes from my data
I haven't loaded my data yet so let's do that actually
MyData equals load_data, the set function we created here,
I'm gonna get a sample as well.
In fact, we can make this sample,
actually this is gonna be a new sample.
Yeah, this is a specific paragraph
and now that's loading properly
so we have our output labels
What we actually wanna do is we wanna
define this dynamically to be,
so this is just taking the labels that we defined here
and then it's just joining them together with a comma
so that way we know we're working from the same labels
when we're using this prompt
so if we ever wanna change these labels up here,
it'll be reflected later, prompt.
We're rendering areas, rendering it as marked down
and then we're also passing those labels into this
QuestionAnswerFormatter
so which is gonna do the hub for us,
of extracting them from the final prompt
and then there's imprompt
and then this basically just defines
what happens if there's an empty result
and it automatically sets this stuff up for us
and then we can just run it, which is
we set the prompt, which is this, the mini batch size,
which is how many labels it does at one time
and then on arrow and then we run with the runner
and the sample
and we're just gonna say, just see,
look at the last five.
Okay, we have an issue here.
The issue just slide.
Yeah so this is the main issue that we had is we wanted
to get the values of the outputs out before.
Yeah, here we go so this run
and we can see the inputs, see the transactions,
and then the outputs
so that prompt is working,
we can check the accuracy of this prompt
so this is doing 85% accuracy
and so it's got 85% of these labels correct,
Which is really helpful to see
and now we have like a baseline,
we can start to optimize that for.
If you're interested in seeing
what the actual prompts look like when they're compiled,
we can actually get that and see what the post
and we're just gonna get the first one.
Here we go so this is what the prompt looks like
We put in the fewshot examples, got the instructions,
and then you can see the output labels.
It's what we've told it to use
and then we've given it these inputs.
Cool so that's how that works
We just want to try and improve this prompt in some way
so there's a few different things we could test.
We could test whether, you know, having a different number
of fewshot examples helps
We could test whether change the instructions helps
or the output labels, whatever we want
We could add extra sections to the prompt,
but we can, we can do a pretty easy test here,
which is by adding in the operators
from the recent samples to the AB test prompts,
which is search EnumerativeSearch operator
We also need the search app,
which is one of the search operator's gonna tell us
to get one of those of the prompts that we're testing
and then put,
and which the convention is, we create like a prompt space,
which is all the different instructions
that we need in order to create the prompt
so ours is gonna be pretty simple
We're just gonna do one of,
and then we're gonna have two different prompts here
so yeah, here we go
We have the one that, this is the instructions that we had
before and then this is the same thing,
but with just, this is very important to my career,
I appended to it instructions
We'll get enumerated each time when we do one of,
so it's gonna test both
and then we've got everything else set up, same you had
before, except, so I'm just gonna copy this, except
you know, now the instructions is coming from this one of,
makes sense.
Okay and then just gonna get returns,
so now we're just returning from this function
We're telling the output, got the mini batch size stuff.
Okay, so we have this ready
and now we can run our AB test
so the way that we run our AB test
now that we've got this setup done
is we just create a sample
and get like 25 and then we create a searcher
and we set up the enumerative search into that
We need the run of first
and then the labeling_prompt_space,
and then the accuracy true, the evaluation at true,
so those, what this is gonna do is gonna run this
enumerative search, which will basically just go through
with the run, iterate through the labeling prompt space,
in this case it was just gonna AB test these two
instructions and then,
it's gonna measure them based on the accuracy.
Now, let's do y_pred, searcher.fit_transfrom
sample and then show_report, let's run this.
Okay, yeah so that was cached already
so that ran pretty quickly here,
but you can see, there we go
This one here, this prompt with very important to my career
I had zero errors like the other one,
except it got slightly better accuracy, 84%
so that's a super simple way to AB test.
The optimizer I found is a little bit buggy,
but I'm gonna copy and paste
an example here that I did get working
The example in the documentation
is a little unclear to be honest,
I ran into a bunch of issues
but essentially the way this works is similar to this,
I actually couldn't get it to work
with the method prompting structure I had before,
but if I set up basically the same prompt here,
but instead they seem to use paragraphs instead of sections
and then it's still got the one of as you can see here
and then you've still got these things that you
should like fit output labels, et cetera
and they also it doesn't seem to work
unless you have render as raw
and then you have this like other formatter,
which is the plain formatter
and then we're getting the labels from
the training data in this case
so we're setting up a class initializing the training data
and then we're getting the labels
and then we're setting up slightly differently here
I would just copy and paste,
if you're gonna do an optimization, I'd copy and paste this
and then just change some of the instructions
and kind of use the same form
I also found that in the documentation it recommends
so this is raise,
I would just follow this exactly if I was you
but then the way it works, I'm just gonna demo this for you,
pull in the data and then you create the training data
In this case we're getting sampling 10 from my data
and then you create these mutation operators
and you just tell it what it is you're trying to mutate
It's just gonna rewrite the prompt
That's basically what they mean by mutations
Gonna rewrite the instructions and paraphrase
and then we'll see how that works
so if we run the beam search,
basically just passing in all these different things,
the runner, the mutation operators accuracy,
and these parameters,
don't worry too much about what these do
This is the report after actual report, right?
It shows the different things It's traded blank one,
try rephrasing it in different ways
and then you have the different objectives
so it seems to get 90% on some of them, which is great
and then you can just here print out the best prompt
so it's a little bit more accessible than DSPy I find
but the optimizer I found it's really buggy
Sometimes you run it and you get the objective of zero
It just really depends on what you do and what you get
Yeah, I would say it's still pretty early for optimization
I probably wouldn't use it for that,
but I imagine by the time you watch this video
that they would've improved it
and I found it is very elegant
and easy to use for AB testing
so I think there's some promise here.