-: Okay, let's walk you through the Vision Prompting Guide,
which is how to prompt a vision model
and just really showing you what they're capable of doing.
So, vision models are typically called multimodal models.
That means they have multiple modes of input.
Not just text, but you can input images as well,
and I'm sure there's gonna be more coming
in terms of inputting video and inputting audio, et cetera.
So, one of the big breakthroughs I think in recent years
is that ChatGPT can see now.
So, OpenAI just, they released that a few months ago
at the developer conference at the end of 2023,
and now it's pretty easy to have AI look at an image
and tell you what's in it.
Back in 2014 it was so impossible to do that,
that there was a joke on the internet on XKCD
that if you needed to take a picture
and see if the picture contained a bird,
it would be like a five-year research project
with a team of PhDs, right?
So it was virtually impossible,
whereas now it's pretty easy.
You just ask ChatGPT.
Which multimodal models are available as well as ChatGPT,
which uses GPT-4 Vision?
There's also CLIP by OpenAI.
It's an open source model,
and that was released as part of Stable,
it's actually what Stable Diffusion was originally built on.
It's a very simple model, though.
It can only really tell you what's in an image
or give you captions of an image.
Gemini 1.5 Pro by Google is multimodal.
It also accepts video, which is really cool.
Claude 3 by Anthropic now accepts images too.
And then there's a couple of other open source I'll turn to.
So LLaVA 1.6 by Meta is open source.
That is a fine tune of the Llama 2 model,
so that accepts images as well as text.
And then there's also Qwen VL, which is by Alibaba.
So, what can it do?
Here's a an example that really blew my mind.
I uploaded an image of a Twitch stream
of the game "World of Tanks,"
and it could pull out lots of information.
It could detect different objects.
Is there a tank in this image?
It can read text from the image as well.
It can understand overall layout.
UI can give feedback on that
if you are prompting it for that reason.
It can also understand what's happening.
Like, it can explain what's happening in the scene
based on that image, which is really exciting.
What I'm gonna do now is apply
the five principles of prompting to vision prompting,
because one thing you'll see
is it doesn't matter what the AI model is,
essentially there's always
these same five principles that apply.
So, everything you've learned for text and image prompting
will also work for vision as well.
First principle was give direction,
and that's also true here.
People have found that by pointing to specific objects,
literally annotating an image, like drawing a circle
around the thing that you want to ask about,
that works really well.
And you can combine that with traditional ML
and image segmentation techniques.
So you could draw these on programmatically.
Identify if there's a bottle in the image.
Draw an arrow next to the bottle programmatically,
and then query GPT Vision
to ask your question of that bottle.
Specifying format also works.
So JSON mode works here as well, which is really great.
You can get structured text from an image.
So here's an example of reading a driver's license
and pulling out the important information,
and that's really exciting.
There's also the concept of providing examples.
So it can actually accept multiple images.
It doesn't need to be just one,
and it's in the chat format,
just like any other kind of chat that you have with ChatGPT.
Here's an example where they got it
to correctly read a speedometer,
which is something it fails at,
where they just provided a few examples first.
So, the first two there,
two first two images,
they manually wrote the response,
and then the third one they just included the image
and then GPT-4 was able to write that response there
and get it correct, which is really good.
Evaluating quality is just as important with vision
as it is with any other LLM work that you're doing,
because it sometimes gets things wrong.
And so it does a pretty good of recognizing
who these people are.
They're famous researchers,
but the bounding boxes it draws around them
are a little bit off.
Like, you can see it's missed an Andrew Ng, right,
Andrew Ng here with the boundary box.
Don't fully trust it for every task.
You need to evaluate quality,
just like you do for anything.
Division of labor works here as well.
So, if you divide a task up into multiple prompts
or if even within the same prompt,
if you get it to think step-by-step
by prompting it to figure out,
"Let's count these apples row by row,"
it does a much better job
of getting the exact correct response, right?
You can see here where we didn't ask it to do that.
It didn't count the right number of apples.
Cool.
Now, I think the biggest thing
and the most exciting thing for me is
temporal reasoning.
What I mean by that is it can actually understand
what's happening in a sequence of images.
Here's an example where it just showed four pictures
of someone doing a pushup.
They go down, and then they go up again
and you can actually ask it,
"What's this person gonna do next?"
"They're gonna do a pushup."
It actually understands the sequence of images here,
and it could reorder the images as well
if you put them out of sequence,
which is again very exciting.
Now, this is really different.
I think this is the thing
that you couldn't do before with traditional ML
'cause it didn't have a reasoning engine behind it.
You could identify that there are cameras in the image,
you could identify a person in the image,
but it couldn't really figure out,
like, what that person is doing from frame to frame,
and I think that little glimpse of intelligence,
that is really what makes this very special
and worth the exorbitant cost
that comes with using vision models.
I applied this to a project that I worked on recently
where we transcribed the audio for some video,
but we also transcribed the images using AI vision.
So, we got it to provide commentary on the game,
and this is from a Twitch stream.
So, you can see the sequence of images we provided.
These are the timestamps in seconds.
And then this is the commentary here that the AI wrote,
and it's really good,
and you can stitch all this together as well.
But one of the things that we found
is that it can handle maybe up to 30 images at the minute
in terms of the context length,
and it's also very expensive.
So it turned out to be about $6 an hour to do this,
which isn't actually that far off
from paying someone in a low cost country to do it.
We expect that's gonna come down
by an order of magnitude soon.
One thing you have to watch out
is AI models have a mind of their own.
They can't be trusted.
Just like I said before, evaluation is really important.
We found that quite often,
it would refuse to do the request or it'd get things wrong,
so we had to really develop
a lot of good evaluation techniques
in order to make sure this is working.
And it's a real pain
because every time it gets something wrong,
that's costing you real money.
For vision requests,
it could be a couple of cents each, right?
So if you're doing hundreds of hours of footage,
then these problems really add up.
So, we developed a bunch of evaluation metrics.
A few different things that we found were useful
is we had a test set, where we actually knew the answers,
so we could provide a sequence
and then we knew what sorts of things
were in that sequence, and we asked it to identify
whether those things were in the sequence.
And yeah, what we can do then
is get an accuracy score so we can see,
how often did it predict false and it actually was false,
and how often did it predict false,
but it actually was true?
One of the things we found is that it doesn't hallucinate
when something isn't there.
So, it actually had zero cases where something like,
let's give an example.
Is there a tank in this image, right?
Like, it doesn't imagine
there's a tank in the image if it's not there,
but sometimes it would miss something.
One of the things we're looking for is,
was it close quarters combat?
And sometimes there was actually close quarters combat
and it didn't recognize that,
so the accuracy was about 80%
for the things that we were testing it on.
The other thing we needed to do was actually figure out
how close it got with some labeling.
It wouldn't always label the same things the same way.
We had active combat as one of the labels here,
but then sometimes it would come up with a new label,
which is like active combat engagement, which is so close
that it's pretty much the same thing,
and we would treat that as a winning test answer.
This was the reference answer,
and then this is what it came up with.
And because it's so close,
we used cosine similarity on a vector database
just based on this text
that we could get that similarity back,
and we'd count it as a one if the similarity was above 90%,
and you can see some examples where that is.
One thing we expect is that open source models
will bring down cost.
So this is an experiment I ran with 1.6,
and it's a 30 billion parameter model.
It does a pretty good job.
It's slow, though.
At the present, it takes about five minutes
to process one minute worth of video, which isn't great.
It's much slower than real time.
GPT Vision is actually slightly faster than real time.
This obviously will speed up.
This was me running it on my M3 MacBook.
Over time, I expect the speed will increase,
the cost will come down
and be relatively cheap to sequence any sort of video.
But one of the really exciting things I've seen,
which I think will change a lot,
and it's not something I really predicted,
was using this for software development.
tldraw did a really cool feature in the platform,
where you can take your mock-up,
and then it takes a screenshot of that mock-up
and it feeds it to GPT Vision,
and then what you get back is a generated code, right?
So like it literally generates,
design the code for you based on that mock-up.
You just click Make it Real, which is pretty cool.
So, there's gonna be all sorts of new modalities there.
I think now that AI can see,
then maybe typing text into a box
isn't gonna be the main way we interact with AI,
but we'll see.
And one of the other exciting things I found
is that, especially with Google, I've seen some demos here
that it can return structured data from video.
Here is an example of someone who works at Google.
They uploaded a video of American cars,
and then it pulled out structured data
of what cars are in the video,
and I think this is gonna be really big importance,
especially because Google's model
has a 1 million token context window.
So you could potentially take a whole movie
and pull out structured data of which characters appeared
in which parts of the movie.
That could open up quite a bit.
That's the sort of stuff that people like Netflix
pay millions of dollars for people to do manually right now,
which should hopefully be done by AI in the future
for a lot less money.
So, what are the other vision use cases?
I just listed a few here that I came up with.
This is primarily from a paper I'll link to at the end,
but a few that are interesting.
It's pretty good at understanding medical images,
although it's not doctor, so it can at least help you
diagnose what's going on potentially.
It can recognize logos, it can count objects,
it can tell you what's funny about an image.
It's also good at visual reasoning,
so it can infer from visual clues
in the image what's going on.
It can extract text.
It's really good at code architecture,
not just the example I showed you, where we draw a mock-up
and it turns that into front end code,
but also on the backend.
You can draw a mock-up of a data architecture
or a database structure and it can turn that
into Python code.
It can interpret charts.
It can actually pull data from the chart,
so, "Extract all the sales figures from this pie chart,"
could be really interesting.
It can translate language, which is really powerful,
especially when it gets faster than real time.
You'd be able to hold it up, just like you can do
with Google Translate right now.
You can hold it up to a sign and it will,
it can automatically translate that
back into your native language.
It's pretty good at data extraction as well.
You can pull data out the image,
like you've seen examples of.
It can read human emotions and understand them
and infer information from them.
It can evaluate aesthetics.
So, I've seen some people use this in image prompting.
Like, it can evaluate whether something is beautiful or not.
It can score it, which is pretty cool.
It can detect defects.
I imagine this can be used a lot in factories, right?
In order to identify whether something has gone wrong
or the product has got a hole in it
or something needs to be stopped
about the manufacturing process.
And then the final two I think
is where it really gets sci-fi.
Embodied agents being able to walk around as a robot
and it can actually tell you where to go
to get something from the fridge.
So that means it can actually be applied to a robot,
and that robot could then use GPT Vision
to understand where it's moving in the world.
I think that's pretty cool.
But you can also set up a browsing agent.
So a little bit less sci-fi
than a robot walking around your home,
but you can get it to actually operate your browser,
operate your computer and book things for you
or schedule things for you,
so I think that's gonna be really powerful as well.
If you wanted to learn a little bit more
about visual prompting, I really recommend this paper.
It's where a lot of these examples came from.
It's called "The Dawn of LMMs."
That's LM, large multimodal models,
instead of LLM, large language models.
But yeah, I think this is the future.
I think there's a lot of unexpected businesses
that are gonna come out of this that will prove
to be really impactful on our world.
The quicker you can test out this stuff, the better.