In this lesson, we are going to see the steps we are going to follow to create a multi modal LM application
in a using launch.
So these are just the conceptual steps.
And in the following lesson the following video we are going to see this in practice.
You will see in these steps that some of them are familiar to you because they are similar to our traditional
rack technique, but some of them are also new and different.
So the first thing we are going to do as, as always, is to load the necessary modules and you will
see that we are going to use some new modules for this kind of applications.
And you will see that among all these new modules, one of them is especially important is one module
called unstructured.
You will see that we are going to use this module to extract the different multimodal elements of our
document.
So the unstructured module is able to extract the text elements, the table elements and the image element
elements in our project.
We are going to use a sample PDF.
A very simple.
We have created is like a financial statement with text tables and images.
We will extract the text, image and table elements with partition PDF which is one one component of
of the module.
We will see extracted images uh, going into a folder output folder uh, that we create, we will store
the texts, the tables and the encoded images in three, uh, Python lists.
And you will see, uh, how do we encode the images.
We will summarize the text and the tables using, uh chat GPT three.
And then we will summarize the images using GPT four vision.
Finally, we will apply the regular Rag technique using a vector database.
A new uh, component called Doc store.
We will see what this is and a Multi-vector or Multi-vector retrieval.
This is also a new element for us.
You will see that in the exercise.
And finally using this Multi-vector retriever as context this is very important.
We can now ask anything about our multi-modal private PDF file.
Okay.
So these are just the steps we are going to follow in the next video.
And you will see in the next video that you will be able to see the PDF document with the different
elements.
And you will also see a notebook with the code, but also detailed explanation.
So don't worry, this is just a quick summary of the steps we are going to follow.
But in the notebook you have these steps and also detailed explanations about all of them.
Okay, so in the next lesson we are going to see and this uh project in practice super super interesting.
This new, uh, this the next video.