Alrighty.
So after we've implemented the retrieve node, we're going to implement now the document grader node.
So when we enter this node we have in our state the retrieve documents.
So now we want to iterate over those documents and to determine whether they are indeed relevant to
our question or not.
So for that we're going to be writing a retrieval grader chain, which is going to use structured output
from our LLM and turning it into a Pydantic object that will have the information whether this document
is relevant or not.
And if the document is not relevant, we want to filter it out and keep only the documents which are
relevant to the question.
And if not all documents are relevant.
So this means that at least one document is not relevant to our query.
Then we want to mark the web search flag to be true.
So we'll go in later.
Search for this term.
And this is a simple heuristic we're making here.
Alrighty.
So let's go to change package.
And here we want to create a new Python file and let's call it Retrieval Grader.
And this chain is going to receive as an input the original question and the retrieve document.
And it's going to determine whether the document is relevant to the questions or not.
And we're going to be running this chain for each documents we retrieve.
And we'll be leveraging structured output for this.
Cool.
Let's start with the imports.
We want to first import the chat prompt template.
We want to import from Pydantic base module and field and the chat OpenAI client.
And let's initialize the LSTM to be the default with temperature equals zero.
And now we want to create a new class called grade document which is a Pydantic model.
And it's going to have a single field of binary score, which is going to be a string which will be
yes or no.
And in the fields description, we want to write that documents are relevant to the question yes or
no.
So this is important because the LLM is going to leverage the description that we write over here to
decide whether this document is relevant or not.
So this will help the enforcement of the schema.
So we want binary score to be only yes or no.
So we'll take the LLM and we'll use the with structured output method.
And we're going to plug in grade documents class.
Now what is going to do under the hood it's going to use function calling.
And for every LLM call we make we are going to return a Pydantic object.
And the LLM is going to return in the schema that we want.
And it's important to note that the default LLM for chat OpenAI is going to be GPT 3.5.
And if you want to use with structured output, we have to make sure that our LLM supports function
calling or else it's not going to work.
And I highly recommend checking the with structured output function and to really see how blockchain
implements this.
Let's review the prompt we'll be sending to the LLM.
The system prompt you are a grader assessing relevance of a retrieved document to a user question.
If the document contains keywords or semantic meaning related to the question graded as relevant, give
a binary score yes or no to indicate whether the document is relevant to the question.
So now we want to use the chat from template from messages method.
And here we're going to plug in the system message.
And we're going to put here a human message.
And we're going to put a placeholder for the fetched document.
So this is supposed to be the document we want to figure out if it's relevant or not.
And we also want to plug in the original user's question.
And finally let's create the chain.
We'll call it Retrieval Grader.
And it's going to take the great prompt.
And it's going to pipe it into the LLM with the structured output.
And that's it for the chain.
Pretty straightforward.
And now it's time to write something.
And I love writing tests.
And I think it's a super important part of the software development cycle.
I think it's important to note that writing text for JNI and LLM based application is tricky because
first of all, we're relying on a third party and LLM that is statistic.
So the answer that we get are not idempotent.
So every time we send a request to the LLM the answer we get back is not guaranteed to be exactly the
same.
So that's reason number one.
Second because it's a third party then we don't really have control about the availability, the durability.
Availability of that third party application.
And we can get a number of errors like rate limiting like service not available or even internal service
error when we deal with third party.
And lastly, it will cost money because each invocation of the LM costs money, it costs token.
Now we can use cheaper models in order to integrate into our SDLC, or we can even use open source model.
But when we use open source models, we will have to handle all of the operations side.
So the deployment and the scalability and availability of that open source model that we're deploying.
So now I want us to put all those testing problems aside and let's go implement some tests for our application,
which is still better than nothing.
And even though we're not going to run it in a CI CD pipeline, we can run it manually and it will give
us a sanity check to see that our application is working and doing what it's supposed to do.
And lastly, I have to admit that recently all the new models, the top tier models, have become so
much better.
Judging by the quality of the answer, latency and cost, it's always getting better, faster and cheaper.
And I've even seen companies integrating that into their CI CD system.
Despite all the disadvantages I've noted before, and I have to say that there are ways to address and
to mitigate those disadvantages.
But we're not going to discuss them in this scope of this course.
So in the test file let's start writing the tests for this chain.
So I'm going to remove this boilerplate test that we have right now.
And let's start with a bunch of imports.
We first want to import load env to load all the API keys and such.
And we also want to import the grade documents class and the retrieval grader chain.
And of course, we also need the retriever to get us relevant documents.
Let's start with the first test case where this is a happy flow.
So we get a relevant document.
So we're going to call it Test Retrieval Grader.
Answer yes.
And here we're going to take our example question.
So it's going to be agent memory.
And we want to use the retriever to get us relevant documents.
And we're going to get the first documents that we find and take its content.
And this means that this document is going to have the highest score.
And I've accidentally put here the index of one.
You can put here 1 or 0.
All the documents in the beginning of this array should be relevant to this question.
And because we did populate our vector DB with documents that are relevant to agents in memory, then
this should be a document containing some information about agent memory.
And here we are going to invoke the chain and we're going to get back a result, which is going to be
an object of great documents.
Because I remind you, we're using the structured output, and the question is going to be our original
question.
And the document is going to be the document we retrieved.
And finally we want to assert that the result with the binary score is yes here.
Alrighty.
Let's go and run the tests and let's see what we get.
It should pass and we can see everything passed as expected.
And let's reverse the assert as a sanity check to see that the test indeed fails.
And we can see it fails.
So it's working as expected.
All right.
So final run just to see that it's still working.
And if we want we can even run it with the terminal by running Pytest dot dash s v.
We can see one test passed and it's the test retrieval grader.
Answer yes.
All right.
So now let's implement another test which is going to be called Test Retrieval Grader.
Answer no.
And here we're going to have a very similar implementation.
But instead of a question to be agent memory we want to switch it with something not related to the
retrieved documents.
So something like how to make pizza.
So we want to retrieve relevant documents to agent memory.
So we're going to use the retriever with that question from before.
However now when we're going to use the retrieval grader, we are going to plug in a different question
how to make pizza.
And we're expecting that the question here is going to be different than the document, which is going
to talk about agent memory.
And the answer should be no here.
All right.
let's go and run all of our tests again and we can see it all passed.
So our unit tests are working as expected.
Alrighty.
So we know that our chain is working as expected.
And let's now create under nodes a new file and let's call it grade documents.
And here we're going to write our node implementation of grading all the documents.
So to decide whether we want to filter them out or to keep them in.
So we're going to define a function which will receive the state.
And in that state we're going to have already the fetched documents.
We're going to iterate through all the documents.
And our grader chain is going to decide for each document whether it's relevant or not.
If it's not relevant, we're going to filter it out.
And finally, if we have found any document that's not relevant, we're going to change the web searching
flag to be true so we can go and later on search for that query.
So we'll have additional information since not all of the documents are relevant for us.
Alrighty.
So you can pause the video, copy all the imports and the function signature, and then we can continue
and implement it.
Cool.
So now we want to simply print out that we're checking the relevance of the documents.
And let's now extract from the state the original question and the fetch documents.
And we want to create a new list which is called filter docs, which we're going to append to this list.
Every time we found a relevant doc, we'll initialize a boolean of web search to false.
If we find any document that is not relevant to the question, then we'll toggle this boolean into true.
And now let's iterate through all the documents And for each document we want to call the retrieval
grader chain.
And we want to supply it to the original question and the document that we're iterating over.
Of course, we want to take the page content, and this will give us the score whether this is a relevant
document or not.
We're going to take the binary attribute of the score.
So it's yes or no.
And if it's yes, we want to append it to our filter docs because it's relevant to our question.
And if it's not, then we want to toggle the boolean to true.
And we want to continue and we're not adding it into the document list.
Finally we want to update our graph state.
So here we're going to put in the documents.
It's going to be the filter documents.
And the question is going to be the original question.
And we're updating the web search flag as well.
And that was pretty much it.
And we finished implementing the node of grade documents.