Okay.
Self-consistency sampling.
This is something that I use all the time, which I don't actually see a lot of other people using.
I think it's a little bit of a secret trick.
It's actually well known in the academic world, I think, but in production it has huge benefits.
I think people really use it enough now.
What is it?
Self-consistency sampling is sounds complicated.
Really.
What it is, is just generating multiple responses and then choosing either the summary or the aggregate
of those responses, or the most common response, or using some evaluation metric to choose which ones
to to select.
Okay, this is a canonical example of it.
It's instead of generating one response where to this question.
Instead it this one generates three.
And that's fine.
This question is wrong, or this answer is wrong because it got it right two of the three times.
And this is one of the fun things about LMS is that they're non-deterministic, so they don't always
give you the right answer.
If you sample multiple times and then you select the ones that are most consistent, then you can get
the right answer in aggregate, which is really helpful.
Let me show you an example of that to run this.
This is using the async open AI because I find it's much quicker if you're running it 3 or 5 times.
You don't want to add too much latency to your prompt.
You don't want to run them five times in a row.
It's going to take five times as long if you run them all at the same time using an async API.
Then you can get much better results.
All right, this is the system prompt.
Here we're telling it to use step by step reasoning for the questions and then provide the answers JSON
on step by step reasoning and final answers the keys.
And then we're just giving it a question.
And this is, by the way, an async function.
This was just what is needed when you're doing asynchronous calls.
And then we have a function just to extract the answer from the reasoning path.
So what we get is we from the JSON.
We just get this final answer response.
And we're also removing any dollar signs just to make this simpler.
All right.
Now we can run this self-consistency function.
And all that's going to do is take the number of samples and generate that many tasks, asynchronous
tasks.
So this is generating five by default.
And we're going to gather those tasks.
So when all five of them are completed then we're going to get all the answers and then find the most
common answer.
Yeah here we go.
So now we have one answer of 85.
Another answer of 85.
Two others that failed to pass.
And then this one came back in the wrong format.
We could go back and figure out the reasoning steps.
We could go back and see what the answer was and why it failed to pass.
But the nice thing here is that despite all these errors, we still got the right answer 825.
So this is something that is really helpful with with LMS when the answers are non-deterministic, sometimes
it just doesn't follow the format for whatever reason.
So this can be really helpful.
And you know, if you're not getting a consistent response across five you could just change this.
You could say, I want now I want ten samples.
And this is some way that you can essentially trade off quality for cost, because it doesn't really
increase your latency very much to generate five things or ten things at a time.
But it does improve the quality, because now if something is only successful two out of five times,
you're almost guaranteed to get a successful answer.
Whereas if you only run it once, you only have a 1 or 2 in five chance of of success.
So if you scale this up to 10 or 20, then you can increase your odds even more.
And all you're doing is paying more for more tokens.
So this is a really good example of self-consistency sampling.
And you can have this either as a quality step in your AI, or you can use this to generate synthetic
data.
Whatever it is, you can really improve the performance of of your output by, you know, generating
multiple answers.
The other pattern I see, by the way, is not just counting the most common answer, but having some
evaluation metric for checking whether the response is correct.
You could also do a multi generation step.
So in this sample if the answer comes back nothing then you could tell it to go back and generate again
until it comes back with something.
So you can usually mix this with other types of techniques.
And see that's a pretty common use of of this this function.