-: Hi, my name is Jared Zoneraich,
I am one of the founders of PromptLayer,
and we are a platform for prompt engineering.
This is gonna be a pretty quick tutorial
on a very important subject,
which is, how do you evaluate your RAG system?
So I've already built a RAG system,
how do I know if it's good?
How do I make sure it stays good
and how do I get my prompts better?
I've already built a RAG system for this tutorial
that is trained on New York Stock Exchange data
and lets the user ask questions about the data.
So let's get started,
to keep this short,
I've done a little bit of the prompt writing already.
I'll show you, I have something called stock-buddy here,
so this is a prompt in PromptLayer.
It says, "You are an AI stock trader,
please answer the user's question."
User gives a question and then it says,
"I've received some data,"
here's the data, "I will now answer your question."
And we'll get the AI to answer the question that way.
I have also built an AI evaluation system
meant to evaluate how good my answer is.
So basically says, "You're an AI system
evaluating a new data scientist that's hired."
Give a true or false if their answer was correct or not.
So we have a question, we have an answer,
and we have a real answer, and it gives a true or false.
So let's go ahead and create the pipeline to evaluate it,
so we're using PromptLayer to do this.
So we'll call it rag-end-to-end pipeline or call it eval.
So I've actually already created a data set
that has a bunch of questions and a bunch of answers.
I manually figured out these answers,
through my like New York Stock Exchange data.
But you can figure out these answers
a lot of different ways,
but this gives me the ground truth answer
that I could evaluate on.
So let's create the pipeline.
You can, also, if you don't have the ground truth answer,
you can maybe use an AI to figure out
if it's close or if it sounds right,
there's a lot of different methods you could use,
this is just one of them.
So let's build the first step of our pipeline.
The first step, as you know, is retrieving the data.
So I've actually, well, let's call it retrieval-step.
I've actually created like a API endpoint
that serves my RAG data.
So that's the first step.
Let's create our next step,
which is actually creating the AI answer.
So we'll select our prompt template
that I showed you earlier, which is called stock-buddy.
We'll choose the data, which is retrieval-step,
and we'll choose the question,
these are the variables from our prompt as I showed you.
So this is kind of a preview of our eval,
this is an easy way to create our eval,
we get to see the live data.
So here's our AI answer
and then we'll finally add another one,
which is the score, and we'll call it ai-eval.
We'll choose our ai-evaluator,
we'll give it the ai-answer,
we'll give it the question
and we'll give it the real answer.
And it'll give us a true or false, if it was correct or not.
So let's see, there we go, true, true, true.
All of these are correct.
So let's go back to our prompt.
As you can see here,
I've already run it a few times on these versions.
I got 72%, then I got 80%.
So let's jump to it.
We can see it ran it on the full data set.
Show all 100.
And our score is how many of these are true
and how many of these are false.
We can also see some stats,
but this is basically how you test a RAG system,
you need to test it end to end,
you can test each individual part,
but you really need to know, does the whole thing work?
Hope this tutorial was helpful.