In this application, which is still a basic application, we are going to do a very sophisticated task,
which is the evaluation of an application.
So.
We are going to, uh, perform two different kinds of evaluations, and you will see that the second
one is really amazing.
So the problem we want to solve is we want to evaluate the quality of a question and answer application
on a document.
This is something we have made before.
Uh, so the first step is going to be to reproduce the creation of a question and answer application.
We are going to load a document and we want to make questions about this document.
Right.
We know how to do that.
But what we know is that evaluating an LLM application is not easy.
You remember in the previous, uh, chapters talking about, you know, more, uh, theoretical point
of view, we knew that one of the things that is not easy in LLM applications is evaluation.
So this is a problem because the answers can be slightly different from the same question.
So if you ask the same question to ChatGPT twice it is very probable you have two different responses.
Probably these two responses are going to uh mean the same, but they are not going to be in the same,
uh, war by war format.
Right?
So ChatGPT is going to answer the same question in a different way, uh, whenever we ask the same question.
And this makes it very difficult to evaluate the quality of an application, because if you have evaluated
previously a conventional software application, you are used to to have or in machine learning, for
example, we have a this test question and answers, right.
We know the questions.
We know the answers.
So we check the application.
If we have exactly these answers, the application is good.
If the application if the answers are not exactly that we are having a mistake or whatever, right.
So in the case of LLM application, we have a different situation because we are not going to have two,
uh, responses to answers that are exactly the same to the same question.
So a conventional software application evaluation system cannot be applied.
What are we going to do?
We first are going to prepare a list of questions for which we know the answers.
And then we are going to ask them to the application using a chain a long chain a chain for manual evaluation.
So the first thing we are going to do is manual evaluation.
We are going to visualize the results of the application.
And we are going to compare these results with our results.
And the second, uh, evaluation we are going to perform is an automatic evaluation.
So we are going to use the LM application to evaluate itself.
That is the, uh, very amazing part of this application, which is still a very basic application.
So the process now is going to have a two main parts.
The first part is going to be what we already know, which is to build the QA application.
And the second part, which is the interesting part now, is going to be the evaluation in these two
different steps.
First the manual evaluation and then the automatic evaluation.
So if you look at the code in the first part, as usual, we are going to load the OpenAI API key in
order to connect ourselves with ChatGPT.
We are going to create an instance of the Elm, and we are going to load the document.
Once we have the document, we are going to divide the document in different chunks using the text splitter.
And we are going to, uh, use, uh, this operation in order to create a vector database and store
the embeddings we created from this document chunks.
Once we have that, we are going to create a chain in order to, uh, question this document.
Okay.
In order to send questions to this document and get responses.
The interesting thing about this chain we are creating here is this last line.
As you can read here, notice that we have added input key in the chain configuration.
This tells the chain where will the user prompt be located.
Okay, so this is the only difference from the previous basic application.
We are going to evaluate this application with two questions and answers that we already know.
These answers are technically known in the line chain lingo as ground truth answers.
So the first thing we are going to do is we are going to enter these two questions and answers that
we already know.
So we have read the document and we have these two.
My questions and answers that we know for sure.
So the first question is, where is a whole neighborhood of YC funded startups?
YC is Y Combinator is an accelerator in Silicon Valley.
And the answer that we know from reading the document is in San Francisco.
The second question based in the on the document is what may be the most valuable thing for barhait
made for Google?
And the answer that we know is true is the motto don't be evil.
Once we have these questions and answers that we know for sure, we are going to apply the original
chain we created for question and answers application to these questions and answers.
And we are going to compare the results we get with the, uh, answer we already know.
So to the first question.
What is a whole neighborhood of YC funded startups, the LM.
So ChatGPT is answering a whole neighborhood of YC funded startups is in San Francisco.
So it's correct.
This is the answer we know is true.
The second question, what may be the most valuable thing Paul Baguette make for Google?
The result is the answer from ChatGPT, and it says Paul Baguette is credited with creating the phrase
blah blah blah.
Don't be so okay.
The answer is correct.
So the the evaluation of this app has been positive since the app has responded to evaluation questions,
right?
Okay.
So if we do this manually we are satisfied.
But.
What if instead of two questions and answers, we have 2000 or 20,000 or 2 million, right?
It depends on the size of the document we are evaluating.
So the manual evaluation would be very painful.
And luckily we have a way to do this uh using the LM application.
So instead of confirming that manually or ourselves, we can ask the LM to check if the responses are
coincidental with the ground.
Truth answers ground.
Okay.
So what we are going to do is we are going to import a different component, which is the QA eval chain.
So you remember that for the QA application we imported the retrieval QA component.
And we use that to create our chain.
In this case we are going to use a different chain which is the QA eval chain.
And what we are going to do is to apply this chain a to the question and answers we have.
And basically what ChatGPT is going to do with this chain is to read these results, and it's going
to compare each of the answers with the results.
And if they are similar, ChatGPT is going to say correct.
And if they are not, you will say incorrect.
So after doing that automatic evaluation, ChatGPT is telling us that the first result is correct and
the second result is also correct.
So as you can see in this very in this very simple application.
We have achieved two very remarkable things.
First, the manual evaluation of LM applications and second, automatic evaluation.
Remember that this is not going to solve 100% of the problem of evaluating LM applications, because
that is a complex matter, as we will see in in in next chapters.
But I think this is a very good step ahead and is going to start giving you interesting concepts and
tools.