Okay.
So very quickly in this lesson, we are going to see the alternative ways to create multimodal LM applications.
And what is the best one.
Uh, right now in terms of efficiency and in terms of accuracy.
So the key operation in multimodal LM applications is the process we use to convert an image into embeddings.
This is the key operation.
Do you remember what is the key operation in our traditional applications?
It was also a this process right to to the process we use in order to generate our embeddings.
We have different alternatives.
Some of them are more efficient, more accurate than others, etc..
So it's very important to understand very well this step.
Right.
So in the multimodal applications we have the same, uh, the same key operation.
How to go from the raw image to the embeddings we are going to use in our rack technique.
So right now we have.
Mostly two alternatives in order to go from image to embeddings.
The first one is to convert the image into embeddings like that.
And this should be, you know, the the one that we expect to use, right?
Because it seems to be the most direct a and it seems to be the, the, the most efficient, but it
is not as efficient and as accurate today as the second one, which is to create a text summary of the
image, like what we call a caption of the image, and then convert that summary into embeddings.
So the first part is what GPT four vision does.
You give an image to GPT four vision, and GPT four vision tells you what it is in that image.
Do you remember all the use cases we saw in the in the previous lesson?
So this is the part where we are going to use GPT four vision for.
And then once we have this text summary of the image, we then will convert that summary into embeddings
using our regular Rag technique.
And you will see that we are going to use the regular ChatGPT for our Rag technique.
We will see this later.
So we are going to combine both models.
We will use GPT four to create a text summary of the image, and then we will convert that summary into
embeddings using the regular ChatGPT model.
Okay we you will see how we do this in the next lesson.
So we are going to see how to create multimodal applications, multimodal Elm applications with an orchestration
framework like long chain.
So we are first going to see the concepts the steps.
And then we will see the project in practice.