Instructor: So we now heard
about a couple of popular open models.
You learned where to find more models
and you learned why using an open model,
why running it locally may be a good and interesting idea.
What are your options for doing so then?
How can you run open large language models locally then?
Well, there are plenty of ways
of running open models locally on your system
or also on servers you may be rented,
but arguably, the most popular options
are using Llma.cpp, LM Studio or Ollama.
Though it's worth noting that LM Studio
and Ollama are really just wrappers around Llma.cpp.
that make using it more accessible
because Llma.cpp is a very low level solution.
It is a project, a tool you can install on your system,
you can run it on your system, but doing so is a bit clunky
and definitely requires some technical expertise
and maybe also programming experience.
For example, if you wanted to download it,
at least at the point of time where I'm recording this,
especially on Windows, you have to go
to the GitHub repository of this project to release this,
and then from these files, pick the one that's the right one
for your operating system.
But even after doing so, you'll in the end just get
a bunch of tools like the llama-cli
which you have to execute in combination with a gguf file
which is a file format.
You for example, find on Hugging Face
that contains the parameters of a model and some metadata.
And then you would have a very basic AI chatbot experience
in your command line interface, in your terminal.
You could also spin up a server,
but all that definitely is a bit more advanced
and not really aimed at the normal regular end user
who might not have a strong technical background.
That's why on this project page, this GitHub page,
you also find a section
where they essentially recommend other tools
that build up on Llama
or work together with Llama that make using it easier.
And here, you for example, find LM Studio,
which is one of those other tools I mentioned
and one of the main tools we'll focus on in this course
and also, Ollama, which is the other major tool we'll use
in this course.
So as explained, these tools will use Llma.cpp
under the hood.
They will take advantage of it because it is an amazing tool
that delivers amazing performance
for running open models locally,
but they make it much easier
because Llma.cpp itself as mentioned,
is a very low level solution
that's aimed at experienced developers
or people with a strong technical background, I would say.
It does only offer that CLI and server mode as I mentioned.
And whilst it does have plenty of configuration options,
it also basically needs you to understand them
and you might even be forced to build your own version
of Llma.cpp to use it to its fullest extent.
That's why instead in this course, we'll use LM Studio
and Ollama and get all the Llma.cpp benefits
without the pain.
For example, LM Studio is an amazing solution
for normal users.
It gives you a very user-friendly graphical user interface
and no technical expertise is required
in order to use LM Studio.
It makes it really simple to download, configure,
and use open models on your system
and it gives you a nice chat interface that even allows you
to upload and use files
and for example, ask questions about images
or PDF documents.
It does also give you a server mode
besides this graphical user interface
so that you could also run LM Studio locally on your system
and interact with it through code if you were building
an AI-powered application, some internal tool
or some automation you wanna run on your system.
So you can also use open models programmatically
with LM Studio, which is also something
we'll explore in this course.
But what's really great about LM Studio is the ease
of use it gives you whilst also giving you
advanced configuration options
that are there if you need them.
So that will be one key tool we'll explore in depth
in this course.
Another amazing tool would be Ollama.
Unlike LM Studio,
this does not give you a graphical user interface,
but it does give you a quite user-friendly
command line interface.
It's therefore not aimed at users
that have no technical experience at all.
You need some technical expertise,
but you don't need to know how to write code
or be an advanced system administrator
or anything like that in order to use it.
It's still very user-friendly as you'll see in this course.
It gives you that CLI mode
and just like LM Studio, also a server mode
where you can communicate with the open models
that are loaded through Ollama
and managed through Ollama programmatically, which again,
is also something we will explore in this course.
And just like LM Studio, it works out of the box,
but also does give you advanced configuration options
that are there if you need them.
And therefore, LM Studio and Ollama are the two main options
we will explore in this course.
Now I also want to mention one other way
of using open models like the Llama models provided by Meta.
And that would through paid services like Groq.
Now, just to be clear, when using such a paid service,
you of course lose some of the advantages you have
when running these models locally
on your system or your own servers.
But when using Groq for example, you get easy
and quite affordable on demand API-based access
to some of the most popular open models out there.
And especially if you're considering renting your own server
to host and run a model there,
you could instead also consider using Groq
and their API to maybe get access
to the same model at a higher speed and a lower price
because they are able to host these models at scale.
Of course, you lose the full control
and privacy you have when hosting it
on your own servers though,
so it's simply a trade off that may
or may not be worth it for you.
Now in this course, we'll definitely not focus
on services like Groq.
Instead, we will really take a close look at running models
locally with LM Studio and Ollama.
But I still wanted to mention Groq
and services like it as a possible alternative
to running open models locally.