-: Hey, I am gonna show you how to test
open source models on your local computer.
This is a tool called LM Studio.
I found this is working really well for me,
specifically on my M2 Mac,
although it should work on your PC as well.
I'm not gonna go through the download instructions
because I think if you want to install it,
you should just go to lmstudio.ai
and then you can download here.
It should just work.
It worked straight away from me,
which is why I was really attracted to it.
But I would say the discord is really good.
People got back to me straight away when I wanted to know
how to make sure he's using my GPU on my computer.
It's a really active community
and it just makes things super easy.
Cool, now the way this works is you need to search
for a model.
Say for example, if you wanted the new Mistral Model,
which is a pretty small model,
you can run it on your local computer.
Then you can search
and it comes up with the Hugging Face Hub
and you can see like which ones have the most downloads
and this is the Mistral model
with the most downloads here, most comments.
What you're looking for,
all of this will not make sense to you
if you don't know anything about these models,
but let me explain it real quick.
So the Bloke is basically like a guy or an organization
that they gets some funding from, I think A16Z.
But he basically just posts these fine tuned models
on Hugging Face all the time,
all the new models that come out.
He'll put it on Hugging Face
or also train kind of new fine tuned models as well.
So when you see like this one is Mitral OpenOrca
that's trained on the OpenOrca datasets fine tuned.
And then there's like also OpenHermes, yarn,
there's a German one.
So yeah, these are all fine tuned models,
but the main one that we want
to use here is Mistral-70B-Instruct.
That is the chat version of the Mistral model
that was released to open source.
So if you want the normal Mistral model, that's that,
but that's similar to GPT-3
where you have to do a lot of prompting
to get it to talk back to you and full instructions.
Whereas the chat model is much better out of the box.
And typically I just download the GGUF file as well
because it's a different file formats,
it doesn't really matter that much,
but that's the one that I've been using and it works fine.
The quantization stuff over here,
this is basically a technique
to remove some zeros from the numbers in the model.
Like the parameters, it makes it slightly less accurate
because if it was,
if they have say 7 billion parameters,
if you remove a few zeros from each parameter,
it makes that parameter a little bit less accurate
to the fewer decimal places,
but it really improves the size and the speed of the model.
Typically what I do here,
and these are different quantization techniques,
what I typically go for,
I ignore everything,
I just go for the K_M, Q4_K_M
and you can see for every single model
there's like a KM Q4_K_M
and you can see I've downloaded that as well.
And the reason why I choose that one is it usually says
it's recommended if you go for a bigger model here,
so you can see like the Q8 here is seven gigabytes
a little bit bigger
and then there's also like K_S versus K_M,
but I don't think you need
to really think about it too much.
The other thing is that you can click
through to the model card here
if you want to read a bit more
about the model and why it was made.
Cool, the the next thing you need to do,
so actually let me just stop this over here
and just show you what you can actually do.
You can actually run this just like ChatGPT,
but you're running it locally in the browser
or actually in the application and it does a pretty good job
and you can export the screenshots.
The other cool thing is it tells you token speed,
stuff like that.
And basically you just need to load the right model
and then choose the right preset,
here I've chosen and Mistral Instruct
because each of these models,
they require a different way of sending the prompt
and this will basically just send a prompt
in the right format.
So this one it needs to have the system message
and then this INST token and then user message
and then close the INST token
and then the Assistant Message,
it formats that for you.
Otherwise the model doesn't tend to work.
You see ads or this, this is the system message.
You could change this if you wanted but I've kept it there.
But now if you're gonna chat to this
by the way in here, you can do that.
But one thing I would say you want to do is
if you go into Tools,
and then instead of, here we go,
yeah, under Tools or Model Initialization,
there's a couple of options here.
So one is keep the entire model in RAM
and what that would do is basically,
it could make your computer crash
but it will make sure that it goes as fast as possible.
That's one thing you can do.
But the main thing you need to do is click here.
So like I have Apple Metal GPU clicked
and that's gonna basically mean
that it will use my GPU while the model is generating.
So it'll be a lot faster.
Anyway, that's the main thing you need to do.
Anyway if you, but what I'm gonna show you is how
to use it locally in your code.
The cool thing that they did is they made it,
I give you an option to set up a server
and this is in port 1234 with logging and everything
and automatic prompt formatting.
And basically they have mirrored the OpenAI API.
And here I have Mistral Instruct
and I have Mistral Instruct Preset here,
but it's given me an endpoint
like this chat completions that I can call.
They actually also mirrored the OpenAI API.
So all you have to do is change the API base to this URL
and then all of your OpenAI code should work.
However, this is already outdated
because they updated to version one of the API for OpenAI.
So if you've already upgraded then this won't work.
So that's why I'm using this curl request here
'cause that's just more generic,
it's not using the OpenAI library.
So anyway, so that is,
what we're gonna be making use of today.
And now I'll just jump over to the code.
So a couple of things I've done
here is I've imported requests,
I've imported JSON library
and then I've set the URL.
So this is localhost:1234.
So this is running on my own local computer.
And then the format of how you send to OpenAI is the same
as how you send to this model
and the LM Studio's gonna do all the hard part
to split it out for you.
So I'm just gonna run that
and it's gonna format the prompt for me
and send it back and here we go.
Yeah, and this is the response.
So just the same sort of response you get from OpenAI.
So that's pretty cool.
Now the code that I've run here,
this is just how you would test these types of models.
I've just brought in here a time
so I can measure the latency
and then I've also brought in Matplotlib
so I can graph it afterwards.
But the, the key here is that
I test mistral-7B against GPT-3.5-turbo
and against GPT-4.
And for GPT-4 and GT 3.5-turbo,
I'm gonna be calling the Completions API.
But for Mistral it's gonna be called,
it's gonna be exactly the same,
but it's just gonna be calling my localhost.
So I've set up this function here,
the default is 3.5-turbo,
but basically what it does is gets my API key for OpenAI,
puts the prompt in here and the system prompt.
And then we get back the response
and here's the system prompts.
It could actually be the system prompt
or it could just be the user.
So if you set user here
and then you could set the system prompt separately
if you want.
I'm just gonna set the system prompt is
"You are a helpful system."
And you can make that whatever you want.
But this is gonna set in the prompt as a user
and then it is gonna fill in the temperature
and everything like that as well.
But this is my prompt,
this is what I was using before.
And what it's gonna do is it's gonna run through each model
and then it's going to create an object for,
to hold the latency and the response or the responses
and then it's gonna go through 10 times
and call the model 10 times.
It's gonna time it, so it's gonna start the timer
and then end the timer after it gets the response back
and it's gonna add those to the object.
And then basically we're gonna have
then this result object filled in
with 10 calls to each of the models alongside the latency.
So then we're just gonna plot it afterwards as well.
So I'm just gonna run that
and that's gonna create the chart at the end.
Now the reason why you'd want to do this
while we're running this is just that,
there's a few different reasons actually.
One is, if you can get better results
from an open source model
or if you can get at least good enough results
from an open source model with your prompt,
then you might want to be using that locally instead.
Like you don't need the internet to use this,
which is very cool because if you're on flight
or you don't have, you have spotty Wi-Fi,
you can still be using an LLM locally,
which is pretty good.
The other reason why it's useful is that
it could be a lot cheaper, right?
If you are hosting this yourself,
you're just paying for the inference costs,
you're not paying any markup to OpenAI.
So that can be really helpful.
A third is flexibility or freedom.
So if you're building a business
and you're worried about things going wrong in OpenAI,
what happens if they fire (indistinct) again
or some sort of disruption happens,
like their API goes down quite a bit
and so having a backup, I think that's open source
that it can't be taken down.
It's literally just code on your computer
so nobody can stop you from using it,
which I think is really powerful
and you can build a business around that as well.
You can host it on your own servers
and then you never have any issues
with reliability from other hosts.
Especially if OpenAI starts having to curtail some
of the uses of of its API,
'cause it gets pressure from Microsoft or whatever.
So anyways, that's what we're run now
and we can see the difference
and we can see that Mistral,
it was faster than GPT-4 on average.
Let's, so this is latency is how many seconds it took
to get back the response.
And then you can see 3.5-turbo is still a lot faster
and pretty cheap as well.
The 3.5-turbo, this probably cost less than a penny.
So anyway that's pretty interesting.
We could also, because now we have this results object,
we can just see what it came back
with I hear are the responses.
You can see that with Mistral
it was pretty good at following instructions,
although sometimes it does put here,
it put the numbers in here,
number one, number two, number three,
and that's not what we wanted.
Whereas if you look at say,
here are the responses for GT 3.5-turbo, it never did that.
So that is a little bit more reliable.
And then GPT-4 obviously is really good.
The other big benefit of doing
this multiple times is you just see
how reliable it is and how often the same names come.
So pretty cool that OmniFit shoes,
OmniFit comes up for all of them.
OmniFit walkers comes up pretty commonly for GPT-4.
It's actually more deterministic it feels
than the other responses.
So seeing this just gives you a good sense
of what are the differences between the models.
I would say that Mistrals pretty good here though,
especially for a tiny model.
If you think about GPT-4 is like a trillion parameters.
A GPT-3.5 I think is like
something like 50 billion parameters
and Mistral is only 7 billion parameters
and it's still get hangs tight.
Some of these names are even better
actually I think AdaptiFit.
So that's AdaptaFit, which interesting OmniFit Shoes,
but there's also like FootMorph I think is pretty creative
and then FootMate, but that's cool as well.
I don't see that anywhere else.
But yeah, I think it is definitely a contender
and it's worth checking out.
So yeah, test your prompts
and hopefully you get good results.