Okay.
So now let's talk about the beta testing phase.
The main challenge we have and how long chain is going to help us solve this main challenge.
So remember that the beta testing phase allows developers to collect more data on how their LM application
is performing in real world scenarios.
In this phase, it is important to develop an understanding for the types of inputs the app is performing
well or poorly on and on how exactly it's breaking down in those cases.
Both feedback collection and run annotation are critical for this workflow.
This is what the long chain team is telling us.
Both feedback collection and run annotation are critical for this workflow.
Don't worry, we are going to see how feedback collection operates in the practical part of the of the
blog and how to create annotations.
These will help in curation of test cases that can help track regression or improvements and development
of automatic evaluation.
So the main challenge that long a long chain team has identified that the LM app developer teams have
in the beta testing phase is how to process and analyze the feedback of the initial users.
And they have the solution for this challenge, which is to use Lang Smith to filter traces with negative
human feedback to understand the problems behind them.
To use Lang Smith to inspect interesting traces and enter annotations about them, and to use Lang Smith
to span the test data set by adding runs and examples.
Let's see each of these solutions that the Lang Smith team has found.
First, use Lang Smith to filter traces with negative human feedback to understand the problems behind
them.
When launching your application to an initial set of users, it is important to gather human feedback
on the responses it is producing.
This helps draw attention to the most interesting runs and highlight edge cases that are causing problematic
responses.
Okay, so these are again the conclusions from the A long chain team.
They tell us that it is very important to gather human feedback on the responses our application is
producing.
So Lang Smith they continue allows you to attach feedback scores to log traces.
Oftentimes this is hooked up to a feedback button in your app.
Then filter on traces that have specific feedback tag and score.
So remember how chat GPT a is getting feedback from you?
This thumbs up and thumbs down button.
So this is what the long chain team is talking about, right?
This is the positive or negative feedback we usually have from users in an LM application.
You can be much more sophisticated than this, but these are the most common.
So a common workflow is to filter on traces that receive a a poor user feedback score.
And then drill down into problematic points using the detailed trace trace view.
We will see this in action in the next practical lesson.
Right.
But I think the a conceptual explanation is clear.
So one of the things one of the, the, the things we can do in order to understand the feedback from
our initial users is to filter traces with negative human feedback.
Okay.
So we will use Lang Smith in order to do that next.
In order to solve this big challenge, how to process and analyze the feedback of the initial users.
We can also use Lang Smith to inspect interesting traces and enter annotations about them.
Lang Lang Smith supports sending runs to annotation queues, which allow annotators to closely inspect
interesting traces and annotate them with respect to different criteria.
So this is what the Lang Chain team is telling us, right?
We will see how we are going to use annotations when we are working with real applications, but you
get the point.
So these annotations are our own feedback to what we observe in the, uh, Lang Lang Smith traces.
Right.
And remember that in this case, we are not just talking about the LM app developer, we are also talking
about other people involved in the development of this application, like product managers or even subject
matter experts, who probably are not technical at all, but they understand the a the the goal of our
application.
Right.
That's why, uh, the, the Lang Smith or Lang chain team is telling us annotators can be product managers,
PMS, engineers or even subject matter experts.
This allows users to catch users.
Here are Lang Smith users.
So this allows the Lang Smith users to catch regression across important evaluation criteria.
Okay.
Finally, we can use Lang Smith to span the test data set by adding runs as examples.
Remember, we are going to talk about all this terminology in the next block.
What is a run, what is a trace, what is an LM call, etc..
So as your application progresses through the beta testing phase, it is essential to continue collecting
data to refine and improve its performance.
Lang Smith enables you to add runs as examples to data sets from both the project page and within an
annotation queue.
Expanding your test coverage on real world scenarios.
So.
In short, when we are talking about traces, NLM calls and runs, we are talking about different.
Degrees of a detail when we are inspecting or researching what is happening under the hood.
So what we are seeing here is that in the test data set, we not only can add traces, we can also add
runs.
We will see how to do that when we go to the practical exercise.
This is a key benefit in having your logging system and your evaluation slash testing system in the
same platform.
So this is one of the main benefits of Lamb-smith.
So Lang Smith is presented as a full LM ops platform, and even when it is a probably the new kid on
the block where you can find other adults in the room, you know, so there are other applications that
are, you know, very solid and, uh, with more years than Lang Smith.
But the very interesting thing about the Lang Smith platform is that this is developed by the Lang Chain
team.
So, uh, probably you don't have other people with better knowledge about LM application development
teams than the Lang chain team.
So it is very interesting that with this knowledge and expertise that they have from observing LM application
development teams, uh, working with all this knowledge and expertise, they have created this new
LM ops platform.
Okay.
So this is very important for us.
In the next, uh, lesson, we are going to talk about how Lang Smith is helping us.
To solve the challenges in the production phase.