Instructor: So how can you access
locally running Large Language Models
programmatically with help of LM Studio?
Well, as a first step in LM Studio,
you should go to power user or developer
since now it is all about development.
And then there to this developer view,
this developer area here.
And there, you must switch this status
of the LM Studio server,
which is the thing you will be able to access
programmatically from stopped to running.
As a result of doing that,
you'll get a log output down there,
which shows you log messages from requests
that were sent to that server.
You can clear that and you'll see any new messages
as they arrive or as new requests hit the server.
You see the API endpoints, so the URLs you could say,
to which you can send requests to, for example,
get a list of available models
or to generate a new response from a model.
And you see the domain to which you have
to send these requests.
So you send them to a combination of that domain
and then this endpoint.
Now, if you don't know anything about writing code,
then this lecture
and this part of this section won't be for you
because of course, programmatically
using Large Language Models only is interesting
if you know how to write code
or if you at least have a basic understanding of
how programming works.
Of course you can use a Large Language Model
to generate code for you,
but you still need to have a general understanding
of what an API is
and what it means to send a request to an API.
But if you got that, then LM Studio has you covered,
because now we got a running server running locally
on our system only accessible by us here on our machine.
So, not available to the worldwide web
unless you configured your machine
to be reachable from everyone else out there, of course,
which by default won't be the case.
And you then also get some settings you can tweak here
about that server.
Now, for example, the port, which is also reflected here,
to which the requests have to be sent,
you can configure just in time model loading,
which means if a request would be sent for a model
that's not currently loaded in LM Studio,
it would then be loaded on demand.
You can turn this off if you don't want this,
but if you turn it on,
the advantage is that you don't have to
load models in advance.
Of course, requests for models that aren't loaded
yet will take a bit longer
because the model has to be loaded first.
And you can also control
that a model should be auto unloaded if it hasn't been used
for a while, for 60 minutes, for example, as a default.
So these are some settings you can tweak.
But with that, the server is up and running.
And the question then of course is,
how can you use that server?
And the answer, as you also find in the official docs is,
for example, by using the OpenAI SDK, this one here.
And that might be a bit weird
because the idea behind this SDK,
as the name kind of implies, is to use it in conjunction
with the OpenAI API,
so the proprietary models provided by OpenAI.
Which is why when visiting the OpenAI documentation,
you find comparable instructions about using the OpenAI SDK,
just that here it of course makes sense.
But indeed this SDK,
this OpenAI SDK has become pretty much the de facto
standard for interacting with LLM APIs.
Many other providers also adopted it.
For example, Google also allows you to interact with Gemini,
which is their paid models, not the free ones,
but Gemini is the paid LLM
provided by Google through the OpenAI SDK.
And it's the same for LM Studio.
The API that is provided by that LM Studio server
that's spun up by setting it to running.
So the API provided by that server here
is essentially in line with the structure
of the OpenAI API
and can therefore be used through the OpenAI SDK.
The only thing you need to do when aiming to do so
is when initializing that OpenAI SDK,
you have to set the base URL parameter here to this value,
which is the domain
or the URL of this locally running LM Studio API.
Because otherwise, without setting this,
it would try to send requests to the actual OpenAI API,
which is of course not what we want.
Instead, we want to send requests
to our locally running model,
and that can be done through that base URL.
Now of course, it's also worth pointing out
that you could also communicate with
that OpenAI compatible API that is exposed
by LM Studio without using the OpenAI SDK.
You can just send a regular HTTP request
with help of any programming language you want.
It is just an API endpoint
you have to send a request to after all.
But it will be an API endpoint that's in line
with the OpenAI API,
and therefore you can use many of the features
provided by the OpenAI API,
just here with our locally running API
that's provided by LM Studio.
So that's quite convenient,
and therefore we can now take a look at some code examples.