Okay, so this is going to be a super interesting lesson in order to understand what can you do with
GPT four vision and what can you do with your multimodal LM applications?
What are the main use cases of this new multimodal LM applications or multimodal LM Foundation model?
So there have been different ways to classify the different use cases of a multimodal LM models like
GPT.
For vision, we can use this 1A6 main generic use cases.
And we are going to see examples of all of them and applications in different industries.
So the first generic use case will be to identify and describe visual content.
The second to analyze diagrams and images.
The third to provide critiques and recommendations.
The fourth convert image into something new, an image into something new.
Five.
Extract data from image and seeks to solve visual based based tasks.
So let's see examples of all of them and also applications in different industries.
So the first a main generic use case of a multi modal LM models and also multi modal LM applications
is to identify and describe image or.
Elements in one image okay.
So we can use GPT for vision or our LM applications, as you will see at the end of this blog, to describe
to describe what it is in one particular image.
And we can also identify and describe visual content.
For example, we can enter this image to GPT four and ask, can you count the number of children in
the image and GPT four vision is able to answer, can you tell me what they are doing?
And GPT four is able to tell us, okay, so this is a very simple image, but as you will see, it can
get more complicated than this.
Regarding the a second possibility in this initial use case to analyze images.
What we can do, we can analyze medical diagrams and imagery.
We will see that, for example, GPT four vision has a limitation.
Regarding that.
We can analyze tech diagrams and schemas.
We can analyze images and deduce context.
We will see examples of that.
We can do sentiment analysis from an image.
We can do artistic interpretation and we can do data analysis from charts for example.
And these are these are just a few examples on how can we apply this image analysis using the multimodal
LM models or applications.
So instead of one photo or drawing as we presented before, we can load a diagram, a graphic like this
and we can ask GPT four, can you explain this graph and provide insight insights.
And GPT four vision is absolutely able to explain a diagrams.
And you will see it's really amazing what it can do.
We will see more examples.
So when we talk about the use case to provide critiques and recommendations, we are talking about much
more than simply analyze or describe what it is in an image.
Here we are talking about.
To critique an image.
To provide feedback from an image, to provide recommended actions based on one image, or to evaluate
an image statically or from accuracy, point of view, or even a subjective evaluation.
This is super, super advanced.
So let's see some examples.
For example, we can load a weird image like this to chat GPT four vision and ask what is unusual about
this image?
And as you can read in the answer chat, GPT four is totally able to have a subjective evaluation of
an image.
So we are talking about a very, very powerful stuff with many applications in many fields.
We can also convert a an image or to use an image in order to create a different thing, like a storyline,
like a prompt, like a recommendation, or like any other actionable format.
For example, imagine what we can do.
We can.
Make a photograph of an application, a web application, or a dashboard like this one.
And ask ChatGPT for vision A.
Can you tell me how to code a web application like this?
This is amazing.
We can even go further than that.
We can just provide a handwritten draft of the web application we want and we can ask for the code.
So here is the design for a blogging website.
Provide a working source code for the website using HTML, CSS, and JavaScript as required.
Imagine what we can do with these models.
We can also extract data from handwritten text.
Structured data from an image or subjected data from an image.
For example, we can provide an image like this.
This is a pneumatic, a wheel.
And attire and we can ask ChatGPT for read the serial number, return only the number without additional
text.
And as you can see, ChatGPT four is able to read the serial number of the tire.
We can also, uh, extract text from a handwritten note.
So this is, as you can see, an old, uh, manuscript.
And we are asking ChatGPT for vision.
Can you read this?
And here you have.
We can also ask for recommendations.
And this is amazing.
So see here.
This is just a picture of a plant.
And this is what we say.
What is this plan and how should I care about it?
So ChatGPT four identifies the kind of plant, uh, we have in the picture and provides tips, care
tips, uh, for this kind of plant.
It's amazing.
We can also use ChatGPT for vision to start data and use it like this.
We can make a photograph of a traffic signs like this and we can ask.
Suppose it is.
It is Wednesday and the time is 4 p.m..
Am I allowed to park my car at this spot?
This is really amazing.
And boom, immediately you have the right answer from ChatGPT for vision.
We can also do things like solve visual based tasks.
For example, we could use.
ChatGPT for to solve most captchas out there.
The the the problem is that, uh, the, the people from OpenAI, they realize this and they have introduced
some changes.
So right now we cannot use ChatGPT for vision for this, but it could be it was able to do that.
We can also solve other visual based tasks.
We can explain visual situations and we can even make ChatGPT for vision.
Recommend us.
Uh, one strategy to follow based on what we see on an image.
So as you can see.
The amount of opportunities and new scenarios.
These new multimodal LM models and applications open is amazing, so we weren't able to satisfy.
All the opportunities we have with the regular.
LM applications.
And now here you have a total different universe to conquer.
Absolutely available for you to build a lot of new things for many different industries.
So we are in a super exciting moment.
And in this blog you are going to learn how to create a multimodal LM application, and you will see
the number of uses you can make for this new technology.
Okay, so let's see in the next lesson some important limitations we will have to keep in mind when
we are thinking about multimodal LM applications and also multimodal LM models like GPT for vision.
What are the limitations as of today?
This as you know, is evolving very quickly.
So as of today, GPT four vision has some important limitations that you need to know.
So let's see these limitations in the next lesson.