Hey there.
This is going to be a longer video because we're going to implement the self rag paper end to end.
So it's going to be much easier now that we have the structure.
And we have been doing things very similar to this.
We can see the graph that we're eventually going to build on our end.
And basically what we'll do is we'll simply add conditional branching after the generate node.
So instead after generate to go to the end we'll add another layer of reflection.
And in this video we're going to implement the hallucination grader and the answer grader chains.
Both chains are going to reflect on the answer and to determine either whether the model hallucinated
the answer, and if the answer is not grounded in the documents.
And if the answer actually answers the question, the original question.
After we have that, all we need to do is to simply create the conditional edge that will run and trigger
those chains.
And you might be thinking, hey, why I'm not putting this into a note.
A note that will check grounding in the documents, and a note that will check grounding in the question.
Or maybe to have them both into a note and then decide what to do.
And the answer is, of course I can do it if I want.
However, because in this process, we do decide whether we want to finish or we want to generate it
again, or maybe to perform another search, then this heavily implies that we need to choose the next
step.
So obviously here conditional branching sounded more intuitively to me.
Alrighty, let's go to the code.
Alrighty, let's go and create a new file under chains.
And we want to call it Hallucination Greater.
Now in this file we're going to implement a chain that is going to determine whether the answer we get
back from the LM.
The generation is grounded in the documents.
So let's go and start with the imports.
We want the chat prompt template.
We want to import base model and field from pedantic and runnable sequence for type hinting and chat.
OpenAI.
Let's go and also initialize our LLM.
And we're going to do the same trick with the structured output.
So we'll create a class called grade hallucinations which inherits from the base model.
And let's give it a quick description that it's a binary score for hallucination present in the generated
answer.
Now it's going to have only one attribute of binary score which is a boolean, and the description is
that answer is grounded in the facts, yes or no because we set the binary score type hint to be boolean.
Then the length chain output parser eventually is going to Who cast the LM answer into a boolean.
All right.
So now we want to take the LM.
And we want to use with structured output method.
And we want to pass the great hallucinations class.
So basically the answer we will get back from the LM link chain will format it as the pedantic class
of great hallucinations which we created now, which is going to have only one attribute of binary score,
which is a boolean.
All right.
Let's create the system prompt for our chain.
You are a grader assessing whether an LM generation is grounded in slash, supported by a set of documents.
Give a binary score of yes or no.
And yes means that the answer is grounded slash supported by the facts.
Alrighty, now let's create the chat prompt template.
And for that we're going to be using the chat prompt template from Messages, which is going to receive
a list, and it's going to be a list of tuples where the first element is the row, which is.
Here is system, and the content is going to be the system message we wrote about.
And the second message is going to be the role of human.
And the content is going to be a set of facts.
And then we're going to plug in the documents, and then we're going to plug in the LLM generation,
which is what the LLM responded earlier.
We have all the moving parts for our chain.
So let's create the hallucination greater chain, which is going to take the hallucination prompt.
And it's going to pipe it to the structured LLM grader.
So eventually what will happen here.
We're going to get an answer yes or no.
If the answer is indeed grounded in the documents that we plug in.
Alrighty.
So let's go now and write some tests.
So we'll go to test chains.py file.
And here we want to add some tests to the hallucination checker.
So we first need to start with importing the chain and the Hallucination Grader class.
So now we are able to write the test function.
So let's define test hallucination grader answer yes.
And the question is going to be agent memory.
We're going to retrieve the documents.
So now we want to use the generation chain we already have imported.
And in the context we're going to give the retrieve documents which should have information about the
agent memory.
And we're going to plug in the question.
So the answer should be grounded in the documents.
All right.
So now we want to run the hallucination grader chain with the relevant documents we retrieved and the
generation.
And the result should be yes, because we sent valid documents and I see I got an error.
And this is because, um, my import above was before I loaded the environment variables.
So let's just change that and let's now rerun this again.
And we can see the test passed.
All right.
So let's now test the second um case where we do have an hallucinated answer.
So for that I'm going to create a new test function.
And I'm going to call it test hallucinations gradient.
No.
And I'm going to put here the exact content as before.
But I'm going to change the assert into no.
And now in the generation I'm simply going to put here some rubbish.
So let's put here for example a And the string of.
In order to make pizza, we first need to start with the dough.
So let's run it.
And of course we're going to expect that this test will pass.
That is that this is an hallucinated answer.
So we're going to have binary score equals false.
And it indeed passed.
And in line 62 we can remove it because we're not really using the generation.
So that's pretty much it for our tests.
Let's just run everything together.
So let's run all the tests just to have a sanity check that all is working.
So I'm going to select the test chains here and let's rerun all of the tests.
And we can see that all of our tests passed.
Amazing.
Alrighty.
So now let's create a new file and we'll call it Answer Grader.
Of course it's going to be under chains.
And here we're going to implement a chain that will grade the answer and will determine whether this
answer answers the questions or not.
Very similar like before.
Let's start with the imports.
And let's initialize data.
We'll create a class called grade answer.
And it's going to have only one attribute of binary score.
And this binary score let's give it in the field.
The description of answer addresses the question yes or no.
And as always let's go and initialize data.
And let's create this structured LM grader.
So this is the same LM but we use the with structured output and we give it the grade answer pedantic
class.
And now all we need to do is create a system prompt.
And here you are a grader, assessing whether an answer addresses and resolves the question.
Give a binary score yes or no.
Yes means that the answer resolves the question.
And finally we want to create the chat prompt template.
So again we'll use the from messages method.
And here we're going to plug in the system message.
We're going to give it the role of system.
And the next message is going to be a human message with the user's question plugged in alongside with
the LM generation.
So this is the answer the LM generated.
And finally, let's go and create the answer grader chain, which will take the answer prompt.
And we'll pipe it into the structured LM grader.
So eventually we'll get back here an object of the Great Answer class, which will have the information
of true or false, whether it answered the question or not.
So this chain was very similar to the hallucination greater chain.
So nothing new here as far as prompt engineering techniques.
So we simply use the structured output with the pedantic class.
All right.
So now we have the chains that will run the reflection of the generation.
And now we want to incorporate it into our graph.
So let's create a conditional branch.
So I'm going to switch over to graph dot pi here.
And let's go and import the answer grader.
And we want to import Hallucination Grader.
And we now want to define a function.
And this function will be the conditional edge function.
We'll call it grade generation grading documents in question.
Of course it will receive the state and it's going to return us a string.
The string is going to be which node do we want to go next.
All right.
So let's begin.
And first here we want to check the hallucinations.
So I'm simply going to fish out and extract from the state the question, the documents and the generation.
So this is all the information we have so far when we get to this conditional edge.
And let's go and run the hallucination grader chain.
And here we're going to run it with the retrieve documents, with the search or without the search and
the generation.
Now the response we get back has the attribute of binary score.
So let's go and call it the hallucination grade.
And if this value is equals to true then this means that there is no hallucination.
So we got the positive grade anyways.
So let's go.
If this is true then this means that the generation is grounded in the documents.
And if it is the case, what we want to do is to grade the generation against the question.
So whether it answers the question or not.
And for this we'll use the answer grader chain.
And again we're going to have a score which is going to have the binary score attribute.
And if this is equals to true.
So this means that the generation does address the question.
And here I want to return useful.
And I'm not going to return the end node.
I'm simply going to return the string useful because later I want to show you how to use the graph mapping.
And and this is what I'm going to output currently.
All right.
However, if the answer does not address the question but it is grounded in the documents, then we
simply want to return that the answer is not useful.
And if this is the case, then this means that the information in the vector store was not sufficient
to answer the question.
So we want to use external search.
We would like to execute the search tool later anyways.
And if the answer is not even grounded in the documents, then what we want to do is to say that it's
not supported.
And in that case, we want to regenerate it generated again from the documents.
All right.
So now let's create all the conditional edges.
The source node is going to be generate.
And from generate what's going to determine the next node.
Is the function.
Great generation grounded in documents in question.
However now we're going to add the third argument of path map which is a dictionary.
And here we're going to map not supported useful not useful.
The strings that we returned from the previous function.
But because they don't represent a node to go to they don't represent a real node name.
Then we are going to map them into the node names.
So not supported is going to go to generate because we want to regenerate after the answer was not grounded
in the documents useful, we want to go to the end node and return the answer to the user and not useful
will go to web search.
Since the vector store didn't have information well enough to answer the question, but notice that
the strings will return from the great generation graduated in documents and question function.
Then those strings are going to be what is going to be displayed on the edges.
So this is something cool and it really makes our graph even more explainable.
All right.
So let's go now and let's run everything.
So I'm going to go to Main.py.
And let's run this with the question what is agent memory.
Now here we're expecting to have sort of a happy flow.
So the answer is going to be grounded in the documents.
And that is going to answer the question.
So we can see here we have the new graph that was created.
And if we check the logs we can see exactly what is happening.
So the flow we just built and ran successfully.
So the answer was indeed grounded in the documents.
And the generation did answer the question.
So this is very cool.
And we have a very complex workflow right now.
And if we'll go to lengths to our project let's have a look at the last execution.
And we can see all of the nodes that ran.
And we can see at the end after the generate node.
Then we ran the grade generation.
So this is what A triggered the conditional branch that finally gave us the answer.
So you can see here the grade generation grounded in documents in question.
So it ran.
And two queries to Dlrm which decided whether it was the right question that we want to return to the
user.
And finally, if you want to check out the source code, then you are more than welcome to go and compare
it with the code in the repository.
So this is in the branch ten frag.
And you can check out and compare the files and um and continue from there.