Maximilian: Now thankfully,
this memory problem can be solved,
because the goal is not to run these raw trained models
with their float32 or float16 values,
because of course, if you were to ask yourself
whether that runs on your machine or not,
the answer would pretty much always be no,
except for super small models.
That's why we have a solution called quantization
that helps us with running bigger models
on your or mine machine.
So it's a solution that helps us reduce
these memory requirements.
How does this work?
Well, in the end, it's a compression mechanism.
Quantized Large Language Models
are compressed Large Language Models,
and that's achieved by taking the original trained model,
which is not quantized,
and which would therefore take up lots of memory.
And then through a mathematical process,
these parameter values are transformed
to less precise numbers.
So instead of float16, float32,
they are converted to int4 int8 numbers typically,
and int numbers are numbers
without a decimal place.
So something like four, five, 10, minus three,
these would be integer numbers.
And these parameter values are transformed
to such integer numbers so that
they therefore take up way less space per parameter.
And the great news is that the quantization algorithms
that are used for that transformation are so good
that the model performance,
the quality of the model is essentially unchanged,
at least for the commonly used
quantization mechanisms and solutions.
So you get the best of both worlds,
a model that does perform great,
but that does not take up as much space
and memory as the originally trained model would have.
In addition, inference speed may also be better.
Now the good news is that you don't need
to perform this quantization yourself.
So in case you were afraid, you don't need to do that,
you don't need to understand the mathematical process
that's being used for transforming these values.
You just should have heard the term,
and you should understand
that it's about compressing these parameter values
to make them way smaller.
Because if we're talking about half a byte
or one byte instead of two
or four bytes per parameter,
we of course essentially only need around one fourth
of the originally predicted space in memory.
So therefore, this 27 billion parameters model
all of a sudden,
instead of requiring 100 gigabytes of RAM
may only require 25 gigabytes of RAM,
or even less than that, maybe only 12 gigabytes of RAM,
roughly, depending on the quantization mechanism used.
I'll also say right away that the amount of memory
that's required is not just dependent on the parameters,
though they are the most important factor,
but it's also some space that's reserved
for the input and output text of that model
because all that text, the so-called context,
also needs to be loaded into memory
in order to be processed by the model.
But unless you're operating with huge amounts of text,
that will not take up as much space
as the model parameters do, for example.
And that's why typically, when we aim to run open models,
no matter if it's locally on your system
or on some rented server,
we don't use the raw models we find on Hugging Face.
That's also why if I were to click Use This Model here,
I don't see any local options
because this is simply not prepared to be executed locally.
Instead on Hugging Face, for all these models,
if you scroll down a bit,
you'll find derivatives that are based on this model.
For example, fine-tuned versions
that may be better at certain tasks,
and very important, quantizations,
and it's these quantizations that you wanna run locally.
So if I click here, I see quantized versions
of this Gemma 3 model,
which I'm using as an example here.
For example, this one,
which actually technically is a special version
because it was trained by Google with quantization in mind,
which is why it performs particularly well
when running it locally.
But even if you have models
that were not trained with that in mind,
you can use these quantized versions basically
without loss in quality to run them locally
with way reduced memory requirements.
So if you click on such a quantized version,
you get a page that looks similar to what we saw before,
but here, we now actually got
that quantized, optimized, compressed version.
It has the same amount of parameters,
but every parameter is way smaller in size.
Originally it was this float16 value type
that was being used for the parameters for this model here.
Now it's in the end a four bit integer
after quantization, so it's way, way smaller in size.
Now the weird name you see here essentially shows you
that this is a quantized model,
especially whenever you see something like this,
Q4 underscore something,
that is clearly a sign for quantization
because this describes the concrete quantization technique
that was being used, for example here,
that we got four bit integers.
It's also worth noting
that this here is a so-called GGUF file,
which is the file type that's typically used
for these quantized models
because it's a file type
that includes all the quantized parameters.
But in addition, it contains a lot of metadata.
That's just a side note.
So GGUF is typically the file extension
you'll work with when working with such quantized models.
And you will typically work with quantized models
because there is really no reason to use
the unquantized ones.
It's harder, it takes up way more memory,
and in general has way higher hardware requirements.
And as explained before,
the quality is, if at all, only marginally better.
So it's the quantized versions of these models
that we run locally or on our servers.