Okay.
So here you have our multimodal a PDF.
This is a fake financial statement we have created as you can see very simple one.
But we have included a the different elements that we need in order to test our multimodal LM application.
We have images.
We have text.
We have tables, we have diagrams and you will see how impressive.
And the LM, the multimodal LM application is here in the right side of the screen you have the notebook
that you will be able to download.
You are also going to have this this PDF for you.
But a in the notebook you can see detailed explanations and also the code that we are using for this
exercise.
So remember we are going to do multimodal RAC with GPT for vision and language okay.
So this is going to be a multimodal LM application that is using the RAC technique with GPT for vision.
Also GPT three and long chain as orchestration framework.
And chain is doing a very good job right now with multimodal LM applications.
First.
We include here a short introduction.
When do we need a multimodal RAC?
So standard sec with text only files.
But what if we want to use Rag with PDFs or slides that have text, images and tables?
Then we use the multi-modal rag.
Okay.
So these are the different steps.
We have a already talked about this in the previous slide.
But here you have them.
And these are the modules we are going to install.
Remember that A you you will need to remove the pound sign in order to uh execute uh this these cells
in your computer.
We, we use the pound here because we don't want to re-execute or execute it again.
The cell okay.
So we are going the first thing, as always, is to create a virtual environment.
Okay.
So after you have created the virtual environment then you install these modules like chain, OpenAI,
pedantic, Alex ML, chroma db.
This is the vector database type token in order to manage the tokens.
And this is the important part.
We are installing the unstructured module.
In this case, we are installing the full version of the module with this, uh, with this, uh, way
here.
So the unstructured module is the key.
We will use it to extract all the relevant parts of the document, the text, the tables and the images.
And Koroma DB will be our vector store or vector database.
Remember vector store and vector database are a synonymous okay.
So this is important.
In order to use the unstructured modules, you will need to install two other modules the Tesseract
module and the poplar module.
Here you have the instructions in order to, uh, install these modules in your computer.
If you are using a mac, uh, like we do, but also if you are using a different system like Windows
or Linux or whatever.
So these are the instructions, the steps to follow in order to install a tesseract and poplar in your
Mac with homebrew.
And if you are using a different system, here you have the links with uh, instructions to to do that
in, in a different system.
Okay.
So here we say we will use a fake, uh, PDF file with text tables and images.
So this is the name of this PDF file you see here.
And we describe what we are going to do in the code okay.
So this is an explanation of the steps you have here in the code.
Okay.
Uh.
In summary, what we are doing here is.
This is important.
We are telling a long chain the name of the output file where we want to load the images that unstructured
extracts.
Okay, so the unstructured model is, is going to come to this file and is going to grab this image
and it's going to save this image, the extracted image in the output path we are defining here we are
calling the the output path figures okay.
So this a cell is going to create a new folder in our root directory called figures.
And in this figures folder it is going to save the A image elements that the unstructured module is
going to extract.
Okay.
So this is where we configure our this extraction okay.
So we say okay this is the name of the PDF file that you have to a look for in the root directory a
in order to extract your elements.
So you will you will need to have this PDF file in the same directory that you where you have this notebook
okay.
And here is where we are configuring this extraction.
We say okay extract images and then also tables.
And these are the the the limits we are defining etc..
All this you can a learn more about that in the unstructured module documentation etc..
But these are very regular uh configurations okay.
So with this cell we are extracting the table text and image elements of this PDF.
Then a we can see we have a store all these elements in this variable row pdf elements.
If we print what we have here you will see that okay we have the text.
Two text elements, and this one and this one.
And we have one table element.
And we also have if you look in your directory we have created the figures folder.
And in the figures folder you already have the images the extracted images.
And remember that in this particular PDF file.
This is an image.
This is an image and this is an image.
Okay.
So I think we have like eight images.
So then a we want to extract the relevant information.
So we want to store the text the table and the image elements in three lists and in three Python lists.
We cannot send the images as they are.
We need to convert them into binary format and we will use the module base64 for that.
And for the text and table elements, we will loop to add them in their list.
Okay, so we import the base64 in order to convert the images into binary format.
And then we create the the text table and images elements.
Python list.
Initially empty, we encode the images using base64, and then we loop into the text and table elements
in order to create the two Python lists.
Okay, we call text elements the list for the text elements and table elements.
The list for table elements.
Then what we are doing is extract just the text.
We don't want to store the the row classes.
So this is how we do it.
And then we can print the number of table elements and text elements we have.
So as you can see we have one table in this PDF file.
This is the table.
And we have two text elements in this PDF file.
So this is one text element and this is the second text element.
Regarding the images.
They are currently stored in the figures folder.
So if you check in the figures folder, you will see the extracted images.
We will loop through that folder and check if the image file ends with png, jpg, and then we will
provide the full page to to the encode image function to in order to encode it in a base64 format.
And then we will enter the encoding result in the image list.
Okay.
Encoded result.
The following cell may take a few seconds to run.
In my case, it was pretty.
Fast, but consider that this is a very simple and short PDF file.
So the longer your file, the slower this cell is going to run.
When you apply this, uh, this code to different, uh, projects.
So this is where, uh, we are.
Uh uh.
Doing all these operations in this cell.
And if we print the number of image elements, you will see that we have eight elements image elements.
So this is
123456I
don't know what are the other ones.
Probably these ones are images.
So now we can create three functions to summarize the text the tables and the images for the text and
the table.
The functions are very similar, but for the images we are going to use GPT for vision.
So this is the main difference with images in the application.
And it is important we make a note of this to pay attention on how we set the URL.
Let's see this later.
So first we install long chain OpenAI.
Remember to load your OpenAI API key in the dot m file.
So you will have to load your dot m file with your OpenAI API key in the root directory of this project.
Uh, with this.
You are familiar with this cell.
And what we are getting here is the OpenAI key that is in the dot m file.
And then.
We proceed with the creation of three functions to summarize the text, the table and the images, and
you will see that the ones for text and tables are very similar, and also we are familiar with them.
But if you go to the third one to summarize images, you see that we are using.
Chat GPT four.
Okay, so here we are creating two different chains using GPT three and GPT four.
And you see we are here loading both okay.
So in this case the way we proceed in order to get a summary of each of the images is a little bit different.
We are communicating ourselves with GPT four vision, and we are telling you are a bot that is good
at analyzing images.
And, uh, what we want is you to describe the contents of this image, okay?
And this is the important thing to keep in mind the way we are.
Setting the URL of the image.
Okay.
So pay attention.
We are saying okay, this is an image and a remember to include the base64 encoded image, etc..
Okay.
So once we have that we will create a summary for each text table and image element.
So the following cells will take also some time to run.
Be careful because GPT four vision is significantly more expensive than the regular GPT models.
So this exercise is not going to be expensive to run because this is a very simple document.
But once you start using GPT four vision with a, you know, long documents, be careful because GPT
four vision is more expensive.
It also has some.
Sometimes some stability issues.
So OpenAI is saying that they are getting better and better with that.
But pay attention to the cost.
Pay also also attention to the stability of the GPT four vision API.
So in order to save time and tokens, we will only summarize the first two elements of each kind.
Well, this is not true because we can remove this.
Sexercize.
We have done all of them.
So.
Here we are processing the text elements.
What we have included here is just a way to stop at the element you want.
So let's say, for example, you have, I don't know, 20 text elements or 20 images and you are just
running an MVP or, you know, a sample or a whatever.
You are just trying your application or showing your application to, to a quick a review team or whatever.
And you don't want to spend a lot of time or a lot of money with the application, you can limit the
number of elements you can process.
In this case, we are going to process all the text elements we have, which are two and all the tables
we have, which is one, and all the images we have, which are eight.
Okay.
So we do that and after creating the summaries we can now proceed with the.
More or less typical rack technique.
Okay.
So the important thing here, this is the important part or one of the most important parts is that
we are going to use.
The long chains multi Vector Retriever to store all our documents, the summaries and also the embeddings
in a vector database.
So here you have a link to the Multi Vector Retriever documentation page.
In long chain.
You will you will see there interesting information and different examples about that.
We will use chroma uh for our vector database.
And we will use a doc store to store the raw documents, the original documents for that we will use
in-memory store from long chain.
Okay.
This is also a new part uh, for uh for our multimodal RAC we will provide later the ID key.
Now it is just a string.
You will see that in the code then.
So we are explaining here everything that you are going to see in the code here.
Okay.
So these are the steps and this is the code.
So.
Then we create.
We will create the function to add documents to the retriever.
We will create some IDs for our documents using the Uuid for function.
Here you have an explanation on what is this function, what is the purpose of it in the exercise,
etc. then we will create a list of documents using the document class class.
A and then we add the documents to the vector database.
We also store our raw documents in the doc store.
Each raw document has the corresponding UID.
Okay, so as you can see, the connection between the vector database and the doc store is in the.
Ideas.
Okay, so here you have it.
We are importing the ID module.
And then we are using a more or less typical rag with a couple of changes the Multi-vector retriever
and also the in-memory store dog store.
Okay so here we are initializing a chroma vector database and the dog store.
We are initializing the Multi-vector retriever.
And here you have the function to add documents into the Multi-vector retriever.
Now we can add everything to the Multi-vector retriever for text and tables.
Summaries are stored in the vector database and raw documents are stored in the doc store.
But for the images, the summaries are stored in the vector database, but summaries, not the raw images,
are also stored in the doc store.
This is an important distinction.
Here you have the raw documents for text and tables, but in the case of the images, what you have
is the summary.
Okay.
So this is how we are adding everything into the Multi-vector retriever.
And after adding everything into the retriever, we can now retrieve the information.
We can use this approach, but mostly what we want is to use this approach.
So we want to have the Multi-vector retriever as the context.
Okay.
So when we configure our chain in order to get responses from our RAC application multimodal RAC application,
when we define the chain in the context, we are going to use the Multi-vector retriever.
Okay.
This is the main difference.
As you can see, we are using here the Lang chain expression language in order to create this change.
So once we have that we can start uh, making questions about text, images or table elements in our
document.
And you will see here in this examples how precise how accurate this multimodal Elm applications are.
This is amazing.
So we can ask this is a very easy question.
What do you see in the images.
And you know, it responds uh correctly more or less.
Then we can go, you know, into more detail like what is the name of the company?
The name of the company is is right.
What is the product displayed in the image?
So we are asking about this product.
The product is playing.
The image is a modern graphics card.
This is a exactly that a how much are the total expenses.
So we are asking about this number in the table.
And again the response is right.
What is the ROI that the return on investment.
Here you have this data.
The answer is right.
Again how much did the company sell in 2023.
Here we can see that uh, the answer.
Well, the note I included here is see that the previous answer can be seen as a mistake if we look
at the bar chart.
So if we look at this bar chart, you will see that the 22 million sales correspond to the 2024 year.
But it is not very clear because, for example, we don't know if these are sales and we we are here.
The name a.
22.
These are an image.
22 million in sales.
And since these are financial statement, ChatGPT may assume that the financial statement is for the
current year.
So we are talking about the results of the 20.
23 year.
Okay, so probably this is a little bit confusing even for a human.
So this response may reflect the confusion that we are providing here in the document.
So what I see here is see that the previous answer can be seen as a mistake if we look at the bar chart.
But we have to admit that the PDF is a bit confusing about it, since it highlights the 22 million sales
in two different places.
The interesting thing is that later I ask and in 2022 so 2022.
So how much did the company sell the previous year?
And right now the answer is correct considering the bar chart.
So this is the answer.
In 2022, the approximate value represented on the bar chart is 15 units.
It's super interesting.
So the note here says see that now GPT four.
Well actually this is GPT three is taking the right sales data from the bar chart.
Impressive.
So let's change this to GPT.
Well no this is GPT four actually because this is an image okay.
So this is correct.
Very good.
So I, I was very impressed about the performance of this very simple multimodal LM application because
as you have seen with a very small resources and code, because you can you can check the cost of this
application.
The cost of running this application with GPT three and four is really small, and the performance is
really good.
So we were able to get the information, uh, on the images, the graphs and also the tables, of course,
the text.
So it is really powerful because, uh, from now on you can start using, uh, the rack technique with
your private documents, the private data of your company, without exposing all this private, private
images or graphs, you know, to, to the cloud, with, with, with ChatGPT or, or other, uh, models
in the cloud.
So you can have a lot of, uh, tranquility about that security, privacy, etc..
So I think this is an amazing step.
I think this coming year, we are going to see a lot of, uh, multimodal lwm applications and multimodal,
uh, startups.
And I think we are just in the beginning.
So it's super, super exciting the, the moment.
And I think that all of us, we are very lucky to be exposed to this new technology in the moment when
it is so new and and it has so many applications, so very, very excited about this new, uh, multimodal
LM applications.