-: Okay, let me walk you through LangSmith.
LangSmith is really important I think for the workflow
of an AI engineer or a prompt engineer
just because it gives you a very easy,
simple way to trace what's happening with not just prompts
but also chains and then agents.
It becomes increasingly important
when there's lots of different calls going on
and I think that's why LangChain focused on this
as their first kind of major project.
The this sort of tutorial is just a shortened version
of the tutorial on their website, which I'll link to.
But in particular the things to note are that
in order to get it running you just need
to use LangChain like you don't need really anything else.
And there are a few different things you need to just add
as environment variables.
You need this LangChain endpoint which sends,
whenever the tracing happens within LangChain,
it sends this to LangSmith.
And then there's also a project name you can give it.
In this case I'll just put in a unique_id
and then also you just need to set TRACING_V2
and equals to true.
That's all you need to get set up and everything else,
whenever you run anything in LangChain,
it's gonna trace, which is really helpful.
You also need to set up your API key,
which you get from within LangSmith.
And then for this tutorial, the agent uses OpenAI.
It also uses SERP API to do searching.
So I've set this up in an environment variable.
I'm not gonna show you my AI API keys,
but that's what the dotenv file would look like.
And then I'm basically just loading dotenv
with this package, but equally you could
just type it in here if you want.
So I'm just gonna run that.
And then all you have to do is from langsmith import Client
and instead of client, if you want to interact with it,
but you don't actually need this for this demo specifically.
The client is you,
if you wanna pull the results back
or if you wanna set up a dataset,
I'll show you what that means.
Okay, everything else is just LangChain code, right?
We're just setting up the ChatOpenAI API,
we've got the tools
and we're setting up an agent
and we're giving it the ll-math and the serp-api tools.
So it can do, it can use a calculator
and it can search the web
and this is a ZERO_SHOT_REACT agent.
So I'm just gonna run that
and if you guys haven't done this before,
what's really interesting about this code,
there's two different things.
So one is that you have an agent out of the box
that will do the reason and act framework.
So it's gonna, when you give it a task like
how many people live in Canada,
it's gonna first gonna check
whether it needs to use one of these tools
and then it's gonna use it
and then using the output,
then it's gonna check again what does it need to do.
What we want for this to do is to search
for the population of Canada and then give you that result.
And then there's things like this like
who is Dua Lipa's boyfriend,
what is his age raised to the .43 power?
So it would need to search the web
to find Dua Lipa's boyfriend,
but it needs to use the calculator
to do the raising of to the power
because LLMS are famously bad at math
and that's really useful.
The other thing that's interesting
about this code is it's using asyncio.
So this is a good simple example
of how you run multiple prompts all at once.
And this is just a list of prompts
that it's gonna run through
and then gather the results at the end.
This is gonna be so much faster actually
than running the one by one.
So if you're testing a bunch of stuff,
I really recommend you look into Async.
So we're just gonna run this
and you see like took 0.3 seconds, super fast
and that's what the basic at 3.5,
but even for GPT-4, this would run pretty fast.
And then you just wanna wait for all traces,
not necessarily really for this,
but this is just to make sure that everything was finished
with this asyncio call.
Alright, that's run.
And now we get into LangSmith
and we can take a look and here we go.
This Tracing Walkthrough has been run again.
And actually just before I get in this,
these are the different projects that we have
and you can see when it was run
what the latency was in particular
very useful and interesting
and then how many times it was run
and how many tokens you used.
So I've had some pretty big tests
where I've used hundreds of thousands of tokens,
which takes a long time and costs a lot of money.
And the latency was super bad 'cause I was using GPT-4.
It's really useful for me to be able to go back
and pick this out and see what the difference is.
When you click into the project,
it has a really good interface actually
for searching into different things.
You could for example, show me like are there any calls
that had used more than
a certain number of tokens or whatever.
You can also search for any errors that you might have.
And actually in this case, there's an error on everything
and that's because this API key is deprecated, right?
You can identify issues like this.
So I'm gonna go into OpenAI,
I'm gonna put in a new API key
and then we're gonna run again.
And so you can see
this is the type of framework that you're doing.
I'm gonna delete this afterwards,
but if I go up here and I change my OpenAI key to this,
I'm gonna delete this one afterwards
so you guys don't rinse me of tokens
and then I'm gonna run this again and then run this again.
Then we can go back iteratively,
iteratively a change any and anything that we find,
any other issues we find with the prompt
and that's what this is doing,
which is super useful I think.
So this is taking 15 seconds
and if we go into LangSmith,
we go back to the Projects.
You see we've got this new run here and now it's all green.
So that's really cool.
We just identified a problem
and we saw an error and we can drill into it.
Here we had had a problem
and it's saying value error, unknown format.
I'm sorry, I can't provide an answer to that question.
So, this is really interesting
because we saw that one in 10 times.
It's gonna say, "No, I don't know how to do this."
So that's really interesting.
You can flag that and say,
"Okay, there's an issue here
that this happens fairly often."
Then for the ones where it did work,
like you can see what the latency was,
you can see how many tokens you used
and you can also just see what it did, right?
What did it do?
So the input was, what is this?
And then the output is this.
And so how did it get there?
So first the LLM chain said,
"Here's the question that I have."
And the output was,
"I need to perform a division calculation
and then I'm gonna take the action of calculator
and then I'm going to use this for the calculator."
Then it called the tool
and the calculator gave it this answer
and then it fed this back in
and it said, "Here's my observation,
here's what I had decided to do."
And then that was the input and then that was the output.
And then it says, "Now I know the final answer.
The final answer is this."
The actual output ended up being correct.
So you can actually step through this code.
And this is so important when you have agents
because they go off the rails all the time
and being able to see this is is super useful.
I'm gonna click into this one just
so you see, "Who's Kendall Jenner's boyfriend?"
And it looks like it's got the right answer.
So we're gonna dig in.
First it says, "I need to find out who his boyfriend is."
And so I'm gonna search for Kendall Jenner's boyfriend.
The result is that you got this response
and then it doesn't give the actual current boyfriend,
I need to search again.
Now it's saying current boyfriend has adapted its search
and then it's coming back and it's getting this answer
that it was player Devin Booker
and then now it knows the boyfriend.
So it's gonna keep on going through
and now I know the boyfriend, now I can use the calculator.
So this is a really cool way to see
what agents are actually doing
and you can see how long it took
and how many calls as well, how many tokens.
So really useful to be able to do this.
You can also dig into what were the individual calls,
what specifically did the prompt say?
And look at this as the actual data here
that you got from this.
And this is how it decided
between what to do with that data.
So you can really drill into anything you need
and you can see how many tokens are used in each.
That is, I mean, in itself incredibly useful.
But you can also take it from here
and set up things like datasets
and you can do evaluations
as here's like an evaluation framework.
You rating it based on correctness or helpfulness
and then you can actually export this
and start using it for fine tuning as well.
So it's a whole workflow and it's super useful.
Highly recommend you try it.
It is I think still in beta,
so you might need to talk
to someone at LangSmith or LangChain.
That's again access like I did,
but I get on this early
'cause I think evaluation is continually coming up
as the number one problem
that people have when they're using AI in production.
Hopefully you guys can get in early.