Okay, so let's talk about some advanced tips, uh, regarding language myth data sets.
First, how to evaluate your LM application with a test data set in the prompt.
And in the prototyping phase, you will create your own test database.
In the beta.
In the beta testing phase, you will add to that initial database.
Examples of real feedback from your beta users.
Mostly relevant cases.
When the user has labeled the LM answer as thumbs up or thumbs down.
So this is going to be the most frequent, uh, um, use case for user feedback.
The thumbs up and thumbs down buttons.
You see this in the ChatGPT app, right?
So this is right now the most common way to get feedback from the beta users and also from the final
users.
You can use the test data set to evaluate different versions of your LM application with different LM
models, different LM model features, and compare the performance in terms of accuracy, latency,
cost, etc. it is very useful to use the comparison view to compare the performance of different versions
of the LM application with the test data set.
So remember, if we go to the platform.
You remember that in the data?
In the data data set and testing dashboard, if we go to one data set and we click on the test we have,
we have just one, but we have more than one.
We can click here in the compare button to see, you know, different graphics, uh, uh, of the performance
of the different different tests.
Right.
With the comparison view, this is the button that activates the comparison view.
We will see this in more detail in the next lesson.
When we go to to the especially with the professional project okay.
So as you can see, how to evaluate your LM application with a test data set is relatively easy.
How many examples should have the test data set.
So this is a question that the long chain team answer.
They said they langschmidt teams.
So the lang lang lang lang Tain team or the Lang Smith team, they are the same.
There is a typo here.
Says.
That the average test data set has around 20 examples.
When an LLM application development team starts the beta testing phase.
But the right number really depends on each project and how much time and effort they want.
Or they can invest on evaluation.
So I found that this very interesting because I thought that the test data set was going to have like
100 or, you know, hundreds of examples.
But the launching team tell us that the average test data set they have observed has around 20 examples.
So my opinion this is a little bit too too too too low.
But this is what they have observed.
And what they say is that, okay, the right number really depends on each project and how much time
and effort they want or can invest on evaluation.
Of course, this is also another thing to keep in mind.
The bigger the data set, the bigger the cost associated with with it.
But a well, this is right now the information that the long chain team has shared with us, with us.
Another interesting advance tip language myth.
Data sets can be used for more things other than evaluation.
The main use of language myth data sets is evaluation, but some teams have also used them for other
purposes, like few shot prompting or even fine tuning.
Okay, I find this.
Couriers, but I would say 99% of us we are going to use data sets for evaluation.
And finally, offline evaluation versus online evaluation.
Offline evaluation is the current Lang Smith evaluation.
Your LM application is tested against a test data set.
This is the offline evaluation that right now we are having in the LAN Smith platform.
The online evaluation is the next Lang Smith feature they are preparing for us.
Evaluators will run on a sample of your traffic.
For example, evaluate 20% of your downvoted traces with a particular evaluator in production with real
data.
So this is what they are trying to accomplish.
And.
Hopefully we will see this soon.
Okay, so after seeing all these very interesting advanced tips regarding a language myth data sets,
we are going to see how language myth help us solve the main challenge we find during the beta testing
phase.
We will see this in the next lesson.