-: Let's walk you through LangWatch.
So LangWatch is a observability tool for LLMs.
What that means is, you can log your API calls to LangWatch
and then you can see some statistics,
and you can do evaluations in here,
annotation, stuff like that.
The thing that I've been finding really useful
with it recently is the workflow stuff.
And this is brand new, so I think it's still in beta
where if you sign up on their website,
then I think you just need to ask them for access
and it should hopefully oblige.
Tell them I sent you, but I have no affiliation.
I just think that the tool is really cool
and it matches the way that I tend
to do this in Jupyter Notebooks,
except you don't need to use actual code, right?
Like, you can do it in this no-code environment
and it's gonna be a lot quicker.
So, I'm just gonna show you how that works.
And so, here we're gonna do something interesting,
we're rating jokes as funny or not.
This is a common one that I use,
and it's creating the workflow here.
There's a few different components you have to think about.
One is the dataset,
the other is an LLM call or LLM signature.
And then there's also retrievers,
like if you're doing rapid (indistinct) and evaluators,
so drag the evaluator in here,
and we're gonna start getting this set up.
It is just basically like a spreadsheet.
And what we'll do is I'm gonna click in
and edit the dataset and delete that column.
I'm gonna set this up.
The column headers that I have,
so I have a joke, the topic,
and then I have whether it's a funny,
and that's a 1 or a 0.
And then I'm just gonna hit Save.
Actually, we just need to give it a name, Funny jokes.
And then you can just add these in manually
or you could upload from CSV.
So, just gonna go grab this data set from DSPyUI,
It's the thing I made.
Let's see here.
Checked.
And by the way, I think this is like
a drop in replacement for using DSPyUI.
I'm not sure I gonna use the tool I built now.
It's the way I expected it to.
I can see here I have a bunch of jokes, the topics,
and then whether they're funny or not.
I actually have an extra row here,
so I'm just gonna delete that row.
Okay, select dataset.
So now we have the data, we have all the inputs and stuff,
and then we can just drag them across to the LLM call.
So, in this case we're doing gpt-4o-mini.
The prompt is, you know,
tell whether a joke is funny or not.
And then we're gonna input the joke.
We're also gonna input the...
And then we're gonna output the funny one.
And actually, we can test this.
You say, knock.
Who's there?
Terrible joke, great.
Okay, thinking.
It actually thinks it's funny.
(laughs) It's not a great prompt, right?
But you can see the execution cost,
and so you can see here some of the observability features
like what the temperature was
and then there's also, like, annotation.
So you could annotate this and say it was good
or it was not good, et cetera.
We'll dig into that.
That's the general set of how you run an LLM.
But where the magic happens is you can connect your dataset.
What we're gonna do is gonna put the joke
and input to the topic.
And then what we'll do is we're gonna pull the funny column
across to expected output,
'cause you know that's been human-rated,
so we think we know that, like,
whether we think that joke is funny or not.
And then this is the prediction, right?
Gonna check whether the output matches the expected output.
And that's gonna give us a pass score.
We could also pass this on to the end, is how we said.
Now, what we can do now that we've done this
is we can click Evaluate and then just say initial prompt,
and it creates a new version for us,
and we can test it on just the 20% of the test dataset
or the full dataset if we want.
So, gonna save and run,
and it's now gonna go and run this with an LLM,
and you can see that we're failing on everything, right?
0% pass rate.
But this makes it really easy for it to see what went wrong.
You can see just like the one we said before,
it's saying that's a classic pun and it's funny,
blah, blah, blah.
It's not giving us the 1s or the 0s.
So, pretty obvious that when you tell it,
when you say 0, sorry, 1 if funny, 0 not.
And now, if we go up to evaluate,
I'm just gonna call that 1 or 0,
and we should hopefully get some passes now.
Here we go.
So, you can see that it's outputting the right amount
pretty much all the time.
About 87% pass rate.
And we can look into the individual ones that didn't pass
and we can make some value judgements here,
which is really helpful.
Cool, so, that's the evaluation.
I think that's quite helpful.
And then you could, in here, you could say
it's really important here.
All right, could do some prompt engineering.
Great, this is emotion prompting.
And then you could go back and evaluate.
Call it, like, emotion prompting, and save and run.
And now, we're gonna get a different output
and see if that actually makes a difference.
Not exactly the same, 85.7%.
So, you know, that didn't make a difference.
Cool. And now we can do prompt engineering.
We can understand what makes a big difference
to the performance.
Okay, so, I think that's pretty cool in itself.
But then the other thing they just added
was that they connected it with DSPy,
which is why I wanted to check it out,
because I made, like, a no-code DSPy tool.
And DSPy lets you do optimization.
So, for example, BootstrapFewShotWithRandomSearch,
and that does is it adds in a bunch of demonstrations
or a few shot examples to the prompt.
So, I'm just gonna hit Run Optimization here,
and it's gonna take a little bit of time to get,
but it's gonna run DSPy for us in the background.
And then once it the optimization is done,
then we're gonna be able to actually go through it
and choose that prompt at the end at the best version.
Okay, so it's running now.
We can see this is a, you know, version that we have
actually increased the score already,
a 93% of this one, which is pretty cool.
You can see which demonstration's included.
So, this is taken from the dataset
and then it tests whether that works or not.
And then it gives you the score.
So you can see it's trying different candidates
and you can still optimize the few shot examples,
but also the prompter itself.
So, I'll just leave that running for.
And we'll see how well that works.
The other thing is you can actually dig
into LLM calls being made.
So it's making 82 LLM calls each time,
and we can see how much the cost of the step is,
less than 1 cent in order to optimize this prompt.
Here we go, a new best score.
This is pretty good, 100%.
Pretty happy with that.
So that's how that works.
And then you can take this, like,
once it's finished running, it takes a little bit of time,
and you can take this, and then you can apply it.
So you can see here, it says
"Please wait."
But once that's finished, then you can click apply
and then it will swap out the prompt in here,
and it will add the demonstrations into this section here.
And then you can do some prompt engineering
and stuff from that.
So, I thought this was a really cool tool.
It actually really matches heavily the type of things
that I'm doing day-to-day basis when I'm prompt engineering.
So, yeah, I wanted to try it out.
You could also have other LLM as a judge to evaluate this,
which is quite helpful.
I just use the exact match evaluator,
but you could also drag in LLM as a judge.
You could say, we need to make sure that the prompt,
the joke, is this joke funny?
And then it could estimate whether the joke is funny or not.
So, anyway, it's cool.
Hopefully you guys find that useful.
I've seen a lot of success from this
and yeah, hopefully you guys find that useful.