Maximilian: So now to come back to the question,
"Does it run on my machine?"
There are two important parts to that answer.
For one, as I explained before,
preferably you have a GPU that can run the model,
and VRAM, into which the parameters,
the quantized parameters can be loaded.
But if you don't, that's okay too.
Your CPU and your regular system RAM
can also handle these models.
You will be able to run them locally without a GPU too,
it will just be slower.
Inference will be slower.
But of course, the amount of memory you have,
no matter if it's VRAM, which is preferable,
or regular system memory, also matters,
because not all open models can be ran on every laptop,
on every machine, even with quantization.
And the great thing about Hugging Face,
is that if you have an account,
which you don't need for this course,
but which you can create for free,
if you have one, then under Settings,
and their, Local Apps and Hardware,
you can basically share your hardware profile.
So in my case, I have a M1 Max MacBook Pro,
so I added this here as my hardware.
And it then tells me whether that's good or bad.
And you see I'm on the lower end here.
But the great thing is that I got 64 gigabytes
of so-called unified memory,
which is a combination of system memory and VRAM.
So that's pretty good,
and allows me to load quite big models into memory.
But the memory is definitely not as fast
as on high-end Nvidia GPUs, for example.
It's always a trade-off in the end.
But since I added this hardware profile here
on the Model pages of these quantized models,
I got this area here where it tells me whether I can run
this specific model version, this quantized version,
on my system.
So here I get a green check mark, which means yes,
this will fit into memory.
But besides this Hugging Face feature,
LM Studio, which is one of the main tools
we'll use throughout this course,
and which we'll also use to download
and run these open models on our machine,
that tool also has a Model Search,
which we will use in the next section,
and there if you browse models, you'll get a warning
if it's likely too large for your system.
So for example, here, this Qwen3,
235 billion-parameter model
is likely too large for my system here.
This one here, on the other hand, would work.
So, since we'll use that tool anyways in the next section,
you can of course use that indication
to find out which model you can use on your system.
And if you don't have a lot of memory,
you simply have to go for diversions
that have less parameters.
Of course, the model quality,
the quality of the results produced by the model,
in general goes up as the number of parameters increases,
but it's not a linear relationship.
And even these super-small models can be great
for certain use cases,
like, for example, text summarization.
You can also do the math manually, at least roughly.
You can take the amount of parameters you have,
27 billion for example, and you can forget the billion part,
It's the 27 part that's important,
and then if it's a four-bit model,
you can essentially take half
of that parameters number here.
So, 13.5 in case of the 27 billion-parameters model.
Why half? Because a four-bit integer takes up half a byte.
So if you were to multiply 27 billion
times half a byte,
you would end up with 13.5 gigabytes of memory required.
Now, you need to add something on top of that,
because it's not just the parameters that must be loaded,
it's also, for example, the context window.
So the input and output tokens, the chat history, and so on.
And that's why you should add something on top of that.
That's why here we see 17 gigabytes, for example,
as a rough estimate.
And if you then have more system and or VRAM
then those 17 gigabytes,
you can run that model locally on your system.
Of course, assuming there's nothing else on your system
that eats up significant amounts of memory.
It's also worth noting that the model can also be split
between GPU and CPU and VRAM and regular memory.
That's of course not as good as doing it on the GPU only,
but it's better than just having a CPU and no VRAM.
So if you have, like, eight gigabytes of VRAM,
but 32 gigabytes of system memory,
you could still run this model here
because eight gigabytes of the 17 gigabytes
would be loaded into VRAM,
and the rest would be loaded into regular system memory.
That's always important to note.
And of course, if you don't have enough memory,
you'll have to look for some other model.
For example, for Google's gemma models,
which are really good, there also are smaller versions
with four billion parameters instead of 27 billions,
and 1 billion and 12 billion.
And of course, these models require way less memory
in order to be loaded and executed.
And that's the offer, how you can answer the question
whether it will run on your machine or not.
And that's what quantization is all about,
and why it's important.