-: Now let's explore how you can use LangChain locally
with Llama 3 using LM Studio.
Now, the first thing you're gonna need to do
is download LM Studio onto your local machine.
You can get this from lmstudio.ai.
We'll also put the link inside of this ChatGPT lesson.
After you've installed LM Studio,
you can search for a variety of different specific models
that you're interested in.
So for example, I could be interested in this model.
Then what you could do is then
after you've done that, I would also suggest
that you should probably download
the one with the most downloads.
And you can see someone here has put a Llama 3 instruct
and they've done a 64K token limit.
So that one's been quite interesting.
For my sake as well, I've already got the Llama 3 model
and you can get this specifically from Meta AI.
Let me just show you the one
that I would recommend getting for,
which is this Meta AI one.
So you'll want to download this one here.
So there's the Llama Studio,
and you can see I've already downloaded this
at the Llama 3 8 billion instruct.
Now depending upon what level of quantization you go for,
and quantization is basically how the numbers become.
If it's Q4 and Q5 and Q6 and Q8,
you've got different levels of quantization.
So this is eight bit quantization,
you've got six bit K quantization.
So generally the lower the bit number,
you can see the smaller size of the file.
But you should generally go
for a larger model if you can,
if the full GPU offload is possible.
And then that basically means that this model will be fit
inside of the graphics processing unit.
So definitely go for the larger model,
go for a higher version of quantization, pause the video,
download the model, and then come back.
And once you've downloaded that model,
now that the model is downloaded,
go over to the left hand side and click on AI Chat.
And then what you can do is click on a model
and select a model to load.
So I'm for example,
interested in this Meta Llama 3 Instruct.
So then I'm gonna say the configuration being applied
contains different system prompts.
So I'm gonna say accept a new system prompt.
And then what's happening right now is the Llama 3 model
is being loaded locally into your graphics card into RAM
so that you can interact with Llama locally.
So let's say create a really nice story
that talks about how children are having a good time.
And here you go.
So now we've got Llama 3 that's now talking to us
inside of the LM Studio, which is great.
So you've got a nice little story coming back here.
Now this is all great,
but what we wanna do is also take it one step further.
So what I would also recommend you do
is have a look at this.
So let's stop generating your story.
And then if you go over, you've got the playground,
which allows you to do multi-model session.
And you've got this thing called the local server.
And the local server allows us to load a model.
So I am gonna say, accept this model,
that's gonna load in the model.
And then what we can do is we can,
you can see here, I can start a server.
So when I click Start Server,
that actually provides the same API endpoints
that basically the OpenAI provides,
which means that LangChain can now work locally
against the Llama 3 model.
The next step that you're gonna need
to do is download any Jupyter notebooks that are inside
of this specific lesson and install Python
if you haven't got that already on your computer.
So you'll need to install Python
and the various packages installed.
Now I'm gonna show you some of the notebooks.
So the notebook you'll need is this llama3.ipynb.
So once you've got that loaded up,
basically what you're then gonna do is import
LangChain output parsers and some various packages.
And if you haven't installed those,
you'll need to install those.
And then after that, then you can set up some models
that we want to parse back from Llama 3.
So I want, for example, my articles to have sections,
and I want my article outline to contain a title,
which is a type of string.
And I also want it to have a list of article
that section outlines
and all of the sections of the article.
Now I set up my Pydantic output parser
and I get my format instructions from my output parser.
But instead of talking directly to ChatGPT on OpenAI's API,
we can talk to the LM Studio's local server,
which is at local host 1234 and v1.
And then you can see I can set API key not needed.
And I've also set these extra arguments here
with a stop position on triple back top ticks.
And the reason why we've done that is that
because we want it to produce JSON,
we're telling LangChain actors and SEO specialists,
it's very great at generating articles,
but it should only respond in JSON format.
We're parsing in those format instructions,
but we told it we must finish with this format here.
So it has to finish with the end at this.
And when we hit this,
that's when our stop token is gonna be triggered.
And then we have a human message,
I want you to generate an effective article plan
for me on digital marketing.
And we provided the first set of back ticks.
So then what we can see is if we go back to LM Studio now,
so if I go back to LM Studio,
you'll see that the tokens are starting to stream in
and also see that we've got the title
and now it's doing the description.
So it's doing the description key.
And so what we've got here is a JSON object
that is being built up in real time
and it's using the output parsers
that we defined inside of LangChain.
And again, we are hitting that chat completions endpoint
but we're doing this locally.
And now we've finished, let's have a look
and see if we can parse that into a Pydantic object.
So we've now got our outline, so let's have a look.
So we've got the title,
which is the Digital Marketing Article Plan,
and we should also have a dot sections here as well.
And each section has the title of that article.
So we've managed to specifically look at
how do you download LM Studio
as well as installing the Llama 3 instruct model.
And then after doing that,
then what we've done is we've loaded a local server.
After playing around with Llama 3 in LM Studio,
we've loaded that local server,
we've loaded a Jupyter notebook,
and we've changed the base URL specifically
to target local host 1234.
And we've also put in that nice stop model argument,
which basically stops the model as it finishes JSON.
Because what I found is Llama 3 can specifically
start hallucinating more and more JSON.
So it's good to have a stop sequence attached to the model.
Cool.
Hopefully this gives you a bit of an introduction into
how you can start to use LM Studio
to do automations on your local computer.