Maximilian: So let's start simple.
Let's try to find out which kind of hardware we need
in order to run a large language model
on our local machine, on our system,
or on that server you may be rented.
Doesn't really matter here.
So we wanna use a large language model for inference,
as it's also called.
And inference is simply the process
of using a trained large language model,
no matter if it's open or proprietary,
to generate output.
So we're done with the training phase,
we've got our model,
and now we wanna use it to generate output.
And that's of course exactly what's going on
behind the scenes
when you send a message to ChatGPT
or one of those other AI chat bots.
That message is processed by the ChatGPT application
and then sent to the underlying AI model
to generate new output
based on that message.
That's inference,
it's the trained model generating output
based on some input, based on your prompt.
So in order to perform this inference,
this trained model must be executed somewhere,
must be hosted somewhere, you could say.
So in case of ChatGPT,
that would of course be the servers
owned or rented by OpenAI.
When talking about open models,
like the ones you find on Hugging Face,
you could either also find some paid hosting provider,
or what is the focus of this course,
host them locally on your system
or on servers owned by you.
And therefore, this hosting system
of course needs to meet certain hardware requirements.
For example, it would be great
if the underlying machine that runs the model
has a good GPU,
but you don't need that.
You can also follow along with this course
and run large language models locally on your laptop,
your system,
if you don't have a GPU at all,
leave alone a good one.
You can also run them on the CPU,
it will just be slower.
Now, why is that the case?
Because these models,
when they receive some input,
need to perform many calculations in parallel
in order to produce that output.
And it turns out that GPUs
are amazing at performing many calculations in parallel.
A CPU, on the other hand,
can switch between different tasks,
but can only process them in sequence in the end.
That's why GPUs are preferred
to handle all these computations
that need to be performed in parallel
when performing inference.
So in an ideal world,
we have a great GPU,
but a CPU will also do.
In addition to a GPU or CPU,
preferably a good GPU, as mentioned,
we also need enough RAM or video RAM preferably.
Now, video RAM is the memory
that's directly attached to the GPU.
So if we have a GPU,
we typically also have VRAM,
and that's preferably
because the GPU has more direct and quicker access
to the VRAM
than to the regular system memory.
But again, if you don't have a GPU,
regular RAM, your regular system memory,
will also do.
Why do we need that?
Well, because the model,
all its parameters essentially,
and also the context window,
which essentially is the chat history,
your input messages and the output messages,
all that must be loaded into memory during inference,
and it must stay in memory.
So you must have enough memory,
preferably VRAM,
to load the entire model
with all its parameters
and the entire context window into memory.
Now, if you don't have a lot of memory,
that's not necessarily a problem,
you just won't be able to load the bigger models,
but you will be able to load smaller models.
And we'll soon also learn about a technique
that reduces the hardware requirements
of all those models
so that even the bigger models
can be ran and can be used on normal laptops,
especially high-end laptops.
You don't need a supercomputer.