So I call it the practical video.
But in fact we can call this the detailed video because here is where we are going to talk about how
Lang Smith helped us solve the challenges in the prototyping phase in detail.
The practical video will be when we apply all these solutions in a real scenario, and that would be
a later.
So remember the prototyping phase involves quick experimentation between prompts, model types, retrieval
strategy and other parameters.
Okay, so in this phase we are going to test different prompts.
We are going to test different model types.
But well you know that probably we are going to stay with OpenAI.
But we can also test you know different different models.
And also we can also check, you know, different retrieval strategies.
Right.
We have been talking about the rack technique during the bootcamp in in detail.
So we will also a pay attention to this thing, the ability to rapidly understand how the model is performing
and debug where it is failing is incredibly important for this phase.
Okay, so these are words of long chain in their blog post talking about the launch of Lang Smith,
etc. but in this blog post, they are unveiling everything they have learned about the process the cycle
of LM app development.
Right?
So regarding prototyping, they are telling us, okay, this is a how launch.
The launching team understands the prototyping phase and this is where Lang Chain helps, uh, the LM
app developer teams.
So we talk about the first challenge when things go wrong during the prototyping phase, how do we identify
what is failing.
And I told you what we do.
We have Lang Smith tracing enabled when developing a new LM application.
Okay.
We will see what is Lang Lang Smith tracing.
We will see in the next blog we are going to talk about terminology.
We will talk about traces, LM calls, runs etc. etc. but now I just want you to stay with me with the
conceptual approach, with the, you know, high level explanation.
So when developing new LM applications, the Lang Lang Lang chain team suggests having Lang Smith tracing
enabled by default.
It isn't necessary to look at every single trace.
However, when things go wrong and unexpected end result infinite infinite agent loop, slower than
expected execution, higher than expected token usage, etc..
When things go wrong, it is extremely helpful to debug by looking through the application traces.
Okay, so what the Lang Chain team is telling us is like you know what?
When you have a problem in the prototyping phase with something doesn't work, go to the Lang Smith
Tracing dashboard, check this trace and understand under the hood what is happening in order to identify
where the problem may be.
They continue saying Lang Smith gives clear visibility and debugging information at each step of an
LM sequence, making it much easier to identify and root cause issues.
So you will see, for example, we will see this in practice when we are in a RAC application, we can
investigate through Lang Smith each of the stages of the RAC technique.
What is happening there and doing that is going to be.
Much easier for us to identify where is the problem we may be having.
So Lang Smith provides native rendering of chat messages, functions and retrieved documents.
We will see this.
So problem when things go wrong, how do I identify what is failing in the prototyping phase?
You just need to have Lang Smith tracing a naval.
Since day one.
So whenever you have a problem, you go under the hood, which is Lang Smith, and check what is happening.
Second challenge, how do we iterate and experiment in a fast and easy way?
So.
We will see how we can use Lang Smith Playground to iterate and experiment.
And you will see that this Lang Smith playground is, in fact, what Lang Smith used to call the Lang
Smith prompt have or Lang Smith Hub or the hub.
Uh, so this is a, an area in the Lang Smith uh, dashboard, uh, that, that Lang Smith application
platform where we can play with different prompts.
And this is not just a I remember when I saw this playground, I thought, okay, we can prove, you
know, we can try different prompts here.
Uh, but it's not just that.
You can use the trace in the long chain tracing to go from the trace to the prompt playground.
We will see how to do that.
So it's much more interesting than you think.
I, I, I, I was surprised by this when I saw this in action with a professional LMS application.
You will be surprised to.
You will see.
So.
A.
Here we are saying that a create a test data set with Lang Smith.
This is more, uh, like the last, uh, point.
Let's see if we have.
Okay.
So here this is a this is a change in the proper order.
But here you have in the notebook the blog talking about how to use Lang Lang Smith Playground to iterate
and experiment.
So Lang Smith provides a playground environment for rapid iteration and experimentation.
This allows you to quickly test out different prompts and models, so you will see how I mean when you
read this, maybe you are getting the wrong idea.
Okay, so I mean, it's not as grandiose as it seems from this, uh, sentence, but this is very good.
It's very good.
It's a playground where where you really can test different prompts and different models.
Okay, so we are going to see this later.
Remember that Lang Smith right now is in the first is in the first version.
So it is going to improve and we are going to learn where Lang Smith is, is going because they have
shared the roadmap they are following in order to make improvements in the current platform.
So.
You can open the playground from any prompt or model round in your trace.
This is the interesting part.
Every playground run is logged in the system and can be used to create test cases or compare with other
runs.
Don't worry if you now are a little bit confused about runs and and traces and LM cause it's a little
bit confusing.
That's why in the next blog we are going to talk about Lang Smith terminology, and we will define all
these for you in detail okay.
So don't worry.
So we were talking now about the third challenge.
And the third challenge is how to compare the performance of alternative prompts, retrieval strategies
and model choices.
And we told you that the recipe that the Lang Chain team gave us in the article is you need to use the
Lang Smith comparison view to compare the performance of alternative approaches.
So this is what they say about that.
When prototyping different versions of your applications and making changes, it is important to see
whether or not you have regress with respect to your initial test cases.
Oftentimes, changes in the prompt retrieval strategy or model choice can have huge implications in
responses produced by your application.
In order to get a sense for which variant is performing better, it is useful to be able to view results
for different configurations on the same data points side by side.
Lingmerth has invested heavily in a user friendly comparison view for test runs to track and diagnose
regressions in test scores across multiple revisions in your of your application so well.
Langs have has invested heavily in a user friendly comparison view.
I think they are in the first stage so this is for sure going to improve.
But right now it's in a very good shape.
So you will see that with the comparison view.
With the comparison view feature, we can compare side by side different versions of our prototype using
different prompts, different models and different retrieval strategies.
So this is interesting.
And as you see it solves the third main challenge we face in the prototyping phase.
What about the fourth and last one.
How to test the performance of the prototype.
So this is where this goes.
The Lang the Lang Chain team recommend us to use Lang Smith to create a test data set.
So let's see what they say.
While many developers still ship an initial version of their application based on five checks, now
remember we were talking about that.
So I mean many teams are just jumping from prototype to production.
They don't do beta testing or even, you know, a very professional prototyping phase.
So I repeat, while many developers still ship an initial version of their application based on five
checks, we have seen an increasing number of engineering teams start to adopt a more test driven approach.
This is Lang Chain team talking.
So these are Lang chain people that have been observing professional LM app developer development teams
working during months.
And this this is what they have observed.
So they continue.
Lang Smith allows developers to create this data sets, which are collections of inputs and reference
outputs, and use these to run tests on their LM applications.
So we have been talking about this in a previous section of our boot bootcamp.
Remember we talked about.
That we can use a.
Good.
Like confirm a data in order to test our application so we can select, you know, or prepare inputs
and outputs that we know are correct.
And with this list of inputs and outputs we can test our LM application.
Right.
So this is what the test data sets of Lang Smith are doing.
So I repeat Lang Smith allows developers to create test data sets which are collections of inputs and
reference outputs, and use these to run tests on their LM applications.
These test cases can be uploaded in bulk, created on the fly, or exported from application traces.
This is very interesting because as we will see in next, uh, lessons, you can whenever you are reviewing
a trace, you know, or a call or a run or whatever in, in your, uh, Lang Smith platform, you can
see, okay, these are good case.
This is an interesting case for me to have in my test data set.
And you can immediately, uh, include the particular trace or LM call, you know, in the test data
set.
So we will see how to do that.
Lang Smith also makes it easy to run custom evaluations, both LM and heuristic based, to score test
results.
Okay, so we have seen how Lang Smith is helping us to solve the four main challenges LM app development
teams are facing during the prototyping phase.
In this part of the bootcamp, we have seen how long chain works in a conceptual way.
In a next lesson, we are going to see Lang Lang Smith at work with real projects.
Okay, but for now I just want you to understand what is happening when you use Lang chain in the prototyping
phase.
In the next lesson, we are going to see how Lang Smith helps us solve the main challenges we face in
the beta testing phase.