Instructor: So we got LM Studio installed
and set up, how can we now use an open model?
Now, as you learned in the first course section,
you could of course visit Hugging Face
and there, the models catalog and take a look
at which model you personally find interesting.
It's worth noting though that not all models are supported
by LM Studio, necessarily though,
but you can find out by going to the model card
of a model you might want to use.
And then there on Hugging Face, you will find this,
use this model dropdown here.
And if you expand this,
you see the options you have for running it locally.
You might need to configure
the local apps you have installed in your system though.
And for that, you might need to log in
with your Hugging Face account.
By the way, this will not be the only way
of finding new models and they offer if you don't have that.
And if you don't want that, that's fine.
I'll show you a different, more straightforward way
of finding models in a second.
But that would be one way.
So if you, for example, configure that you have Ollama
and LM Studio on your system,
you should see these options here
if you can run this model, like this with LM Studio.
Now here, I see nothing
because indeed, gemma 3 is supported by LM Studio,
but not the raw version of it.
Instead, as you learned in the quantization
and hardware requirements section,
typically when running models locally, you run a quantized,
a compressed version of that model.
So on Hugging Face, we would scroll down to this list here
to find the quantization
that have been created based on this model.
And if we then take a look at some of them there,
eventually we will find some that do support LM Studio
and you could then click on this to actually open LM Studio
and start loading that model.
But that's not the only way of finding models.
The more straightforward
and convenient way of finding models that are supported
by LM Studio is to go back to LM Studio
and then go to the settings and dare to model search.
Alternatively, you could use the shortcut that's shown here,
which might differ for you,
but you can use whatever is shown there for you.
And here, you'll find a list of models
that are supported by LM Studio.
So for example, here, I could use Gemma 3
and then I can choose which flavor, which version I want.
Do I want the big one with 27 billion parameters,
which is bigger to load and requires more RAM?
Or do I want the very small one with one billion parameters,
which runs on low performance machines as well?
And here, I'll go for the middle ground
and choose the 12 billion parameters model,
but it really doesn't matter.
You can go with the very small one too,
and you could also use a different model.
This does not matter. It's just an example here.
It's also worth noting
that all these models here are loading
through LM Studio, through that model search
will normally be quantized.
For example, this here is a Q4 quantization,
which means it uses four bit integer
to reduce the amount of memory this model needs
for running it locally.
So no matter if you're using this model
or some other model when loading them into LM Studio,
you typically use quantized version simply
because they are faster, they take up less space.
And there is no real reason to use the quantized one
as explained in that quantization
and hardware requirement section.
So you can select whichever model you are interested in.
And then here, right inside of LM Studio,
you can learn a bit more about it.
You learn about the number of parameters used by that model,
the number of downloads when it was updated and so on.
You find a link to the model card on Hugging Face,
which you can click, to be taken
to that respective Hugging Face page of that model.
And you also find the model read me down there
with more details about the model.
You can also choose from download options,
though for this model here, I only have one.
And the default option in general is what you want.
Though for some models like for example,
these different versions of the gemma model,
you can actually choose
from different quantization options here.
And the higher this number here is,
the more space and memory that model will take up.
And of course that's not just available for gemma.
If you take a look at some other models,
like the deep seek models here, though,
these are slimmed down versions
of deep seek R1, not the large one.
But if you take a look at models like these,
you also find different quantization options here.
These gemma models here are a bit special
because this QAT model, which I'm using here in this demo,
has actually been already trained with quantization in mind,
which is why there are no different options to choose from,
but for other models, that differs.
And you can choose exactly the version you want.
You also see the size of the download here,
so that will also be the size occupied on your disc,
once the model has been downloaded
because it will be stored on your disc
so that LM Studio can use it now, tomorrow, and so on.
Of course, you can also delete models from your disc though.
So here, I wanna use this model,
I picked it, I'm happy with it.
And now I can click download here
and this will download it onto your machine,
store it in a specific folder,
and then make it available for use here in LM Studio.
And I'll be back once that download finished for me here.
So here for me, the download finished
and I could now click load model to load it into LM Studio
because when you download it,
it's simply stored on your hard drive on your system,
but you then still need to load it to activate it,
so to say, to be able to use it in LM Studio.
Now alternatively, you can also do that through
that model picker here at the top
because once you have at least one model downloaded,
if you open that picker, you should see that model here.
And of course, if you have more models,
you'll see more models here
and you can simply click on a model here to load it.
By the way, if you're not in user mode,
if you're in power user
or developer mode, you will see more options here.
So if I am in power user mode,
I don't just get this downloads window here,
but I also get more configuration options
when I load a model.
So if I click on a model here, which is the one
that's currently already loaded,
but I could reload it, if I click on it here,
I get more options.
For example, I can set the context length,
the context window size that's available,
and we could even add more advanced options as well.
But I'll get back to those later.
For now, I'll go back to the user mode
and just load the model as is with the default options.
Now in the bottom right corner,
you'll also see your system resources usage.
For example, how much RAM
or video RAM the model is taking up
and how much work your CPU has to do.
You can also click on this here to unload all models.
If you click on that model picker, you'll also see
how much memory this model is consuming.
Now here, I am on M1 Mac and therefore,
I have something that's called unified memory.
So there is no difference between regular RAM and VRAM,
the RAM made available by the graphics card.
It's just one RAM here,
which is why I have one RAM number here
and then how much of that RAM is taken up by that model.
If you're on a different system,
it may differ whether the model was loaded
into your regular RAM,
if you don't have a graphics card with dedicated video RAM,
or a graphics card that's not supported,
or whether it's been loaded into the RAM,
made available by that graphics card, the video RAM.
It will simply depend on your system.
In general, if the model is loaded into your regular RAM,
not the video RAM, using it will be quite a bit slower.
So generating the response will take a bit longer
for the model, but it will still work.
Okay, so here, I got this model selected now.
Now of course I can click on create a new chat
and then start chatting with that model.
And just to make it really clear
that this is running locally on our system,
I will disable my wifi so
that I got no internet connection,
and still, I can ask what can you do for me
and send this message to this locally running model.
And then here in LM Studio, you'll get back a response
that's automatically parsed and formatted as markdown.
And that's therefore quite nice to read.
Now I'll re-enable my wifi here,
but we don't need it to use this model, of course.