Maximillian: Now, that's still not all we can tweak
about our models and settings here,
because when you load a model, let's say here,
I wanna switch to that 12 billion model again actually,
which I already have loaded,
but I want to configure it differently.
So, if I load a model here, by the way,
you can always unload it by clicking eject, of course.
But if I load a model,
no matter if I had one or not, then if I'm a power user,
I can tweak some settings.
As a regular user, this is not possible.
As a regular user,
it just gets loaded like this, but as a power user,
there are some options here, as you can see.
Most importantly, you can control the context length,
and that will depend on the underlying model you're using.
For example, these Gemma 3 models
have a quite big context window.
Now, in case you're not sure what context length
or context window means, as mentioned,
all these models operate on tokens,
but they have a maximum amount of tokens they can process.
And that is exactly that context length,
or context window as it's also often called.
You see that here,
this model supports up to 130,000-ish tokens.
And you also see the tool tip here,
the maximum number of tokens the model can attend to
in one prompt.
Now, theoretically,
your chat history could go beyond that limit,
but then not the entire chat history would be considered,
because normally the entire chat history
is sent to the model with every prompt
so that you can also refer back to older messages.
But of course, if you go beyond the context window
with your entire chat history,
then not the entire chat history can be considered.
And if a single prompt goes beyond
the available context length,
then not even that entire prompt can be considered.
Now, what's worth noting about LM Studio,
and later also Ollama, is that by default,
they don't make that entire maximum
context length available to you.
Instead, here, in my case for this model,
the default is around 4,000 tokens,
so way less than the maximum.
By the way, that's around 3,000 words,
because you can roughly say
that one token is around .75 words.
Now, we can ramp this up,
but why don't we always use the maximum?
Well, because the context window also takes up space
in your memory.
Let's say I changed this to 20,000 here
and I leave everything the way it is.
Now, if I load this model,
you will see that for me here,
it takes up around 15 gigabytes of ram.
So, no problem.
I got 48 gigabytes of VRAM, but it takes up around 15.
Now, let me eject it and load it again,
but now with the default setting of 4,000 tokens.
With that, you'll see that once it's done loading,
it's only around nine gigabytes.
So it has to reserve way more space
if I set up a bigger context window,
because it essentially has to reserve all that space
and memory that could potentially be occupied
by your tokens.
So, therefore, of course,
if I ramp this up all the way to the limit to take advantage
of the full available context window size,
you see here for me, for this model,
this goes up to essentially the maximum amount
of RAM I have on my system,
which is typically not what I want,
because now it's really struggling to do anything.
So, you should choose a context length
that makes sense for your use case.
If you know that you are about to process
some big PDF documents,
maybe you need 20,000 tokens, maybe you need more,
but you should not reserve as much as possible all the time.
And if you know that you need to process more tokens
in general than your system can handle,
then of course you might wanna consider splitting your task
into multiple smaller subtasks
so that every subtask can be completed
and you then merge the results together thereafter.
That's what's important to understand
about context length here.
Now, besides the context length,
we also got the GPU offload.
And as you see, if you hover over that,
that's the number of discrete model layers to compute
on the GPU for GPU acceleration.
What this means in the end is how much of the model
should be loaded onto the GPU,
or how much work that needs to be done by the model
should be done by the GPU.
And unless you have a strong reason to change that,
you typically want to keep that at a maximum.
You wanna let your GPU do all the work
because it's much better than that than your CPU,
because the GPU, unlike the CPU,
is really able to handle multiple things in parallel
without constantly switching back and forth between them.
So, typically, GPU offload should be at 100% here.
The context length should only be set to a value
that makes sense for your specific task.