Maximilian: Now when it comes to tweaking those settings,
you can of course also remember them
so that they will be used in the future again.
So if you by default want let's say 10,000 tokens,
you could save these settings as a default
so that you don't always have to change it again
when you reload the model.
But you also have some advanced settings here.
Now, to be very honest, most of these settings
can stay the way they are.
The evaluation batch size is an interesting one
because here you can control the number of input tokens
that are processed at a time,
so simultaneously by the model,
and you could increase this
to improve the performance of the model
at the cost of using more memory.
But I typically keep the default here as well.
But you could experiment with different values here.
The average settings, to be very honest,
are settings I never touch
because the defaults work quite well.
These RoPE settings are about
pushing the boundaries of your model
when it comes to operating with lots of context,
so long context documents or chat histories.
And I typically prefer
to split complex long context tasks
into simpler, shorter subtasks,
because in my experience, that yields better results
than operating on super long context anyways.
Now what is interesting are the flash attention
and the key and V cache settings down here though.
And the idea behind all these settings,
especially when used together, is that they can,
as the tool tip tells us, decrease the memory usage
and also the generation time of some models
depending on how well they support it.
Now, this quantization, this KV cache quantization
is not the quantization
we were referring to earlier in the course
when we explored hardware requirements.
Instead, flash attention and K V cache quantization,
these three settings in the end work together.
As you also see here, flash attention must be enabled
for V cache quantization at least.
And the idea behind these settings
is that the model has to process less tokens
to generate a new output token,
because normally it always has to process
all existing tokens,
including existing output tokens it's generated
to produce the next token in line,
which makes sense because that next token
depends on all these other tokens.
Now, with flash attention
and K V cache quantization enabled,
roughly speaking, tokens generated in earlier steps
can be cached and therefore be remembered,
and can therefore be reused in subsequent steps
to generate future tokens
without having to recalculate them.
Because without this turned on,
the model essentially recalculates all previous tokens
just to derive the next token.
With flash attention and this caching enabled,
it can simply load those old calculations from memory
and therefore derive the next token quicker.
Now, at the point of time where I'm recording this,
this is an experimental feature,
and as this message here tells us,
it may cause issues with some models,
because not all models are great at using this mechanism
or are aware of this technique.
You can nonetheless consider or try turning this on
and simply experiment with the model
with that being turned on
to see whether it makes a difference
and whether it works well for you.
For example, here with that turned on,
if I now ask it to write an essay about a rabbit,
it's quite quick at doing so,
and the text also doesn't look too bad to me.
And it generated those 1000 tokens
at a rate of around 31.3 tokens per second.
Now, if I reload this model,
but I turn off flash attention and K V cache,
and I use that same prompt in a new chat thereafter,
it of course, again goes ahead and generates a text.
And for me here, it generated 1000 tokens
at a rate of only 23 tokens per second.
So quite a bit slower than with flash attention turned on.
And that's precisely the idea behind flash attention
and K V caching.
The idea is to give you faster generations,
but you should carefully check whether the results are good
or if your model is having any problems with these settings.
In that case, you should of course turn off these settings.
But from all these more advanced settings,
these are the ones I would consider turning on
to get faster inference.