All right, let me walk you through Grau, which is a cloud provider for LMS.
And the reason why it's really useful is that it hosts all of the open source models, all the really
useful ones anyway, and it's blazing fast.
So say, for example, here I'm using llama three.
Be it hit submit and boom, it's very fast.
You can see here we've got 1200 tokens per second, which is insanely fast, right?
Even for some of the bigger models as well.
If I go to maybe the 70 B one, just hit submit.
Really fast.
It's still 300 something tokens per second.
Why would you use this?
It's a really quick and easy way to use some of the open source models like you saw, but primarily
I use it for the API.
So rather than trying to figure out how to host all these open source models myself and get it as fast
as as what these guys can do instead, I use this.
So if you click view code, you can see that it follows basically the same thing as the OpenAI API,
which is really helpful in terms of making it easy to to swap in and out depending on what you're using.
I'm just going to bring up a script here.
I'm just going to walk you through this.
That's the that's the script.
And you can see that all we have to do here is pip install grok and grok with a Q.
By the way, don't confuse this with grok with a k, which is the Xe.com or Twitter version of Python.
So a little bit confusing naming there.
But but yeah basically it's grok with a Q and it's a hosting service and model hosting service.
So if you pip install that then all you have to do is load the environment.
In particular it needs to be a grok API key variable.
That's the main thing.
And then you want to set your model.
I've just created this function called grok.
You can call it anywhere you want really.
But the important thing is that you follow the OpenAI naming conventions here.
So you create, you create a client and then you do a chart completion and then create.
And then you put in your system message and your user message.
And then in order to get the text out, it's the same thing as opening choices.
And you get the first choice and then you get the message content.
So here I've put in the system message which is which is basically getting it to tell jokes.
And then I have a user prompt here, which is just the topic of the joke.
So here it's telling a joke about cats.
You can say, there you go working remotely.
And tell a joke about it.
There we go.
So the jokes are actually pretty good.
And this is I think one of the big benefits is that because you're using the open source models, I
find that their performance can be really different from what you get from OpenAI.
And this is really fast.
So just to show you, one of the benefits you can get here is you can very quickly iterate through,
just like all the models.
It took four seconds to run it with one, two, three, four, five, six seven different models, and
some of them were fairly big, right?
Like the 3.2 text preview.
They tend to get these models online pretty quickly.
They also have audio for transcription and they have vision models as well for visual.
Transcription.
You can see here you can compare the different models and see how how good they are relative to these
things.
Yeah.
This is this jerk better than this one.
You can you can start to see whether it's worth it to do a bigger level model, for example, which
shows some trade offs in terms of cost and latency.
But it might be worth it for some tasks.
And here you can see this is a common problem I find with the mixture model is that instead of giving
you the joke, it just gives me some preamble first, and then it gives me the joke, which can be a
bit of a pain.
You can start iterating through these things and see how they work.
So really simple.
It's just as as simple as calling a OpenAI.
But you're getting a really fast response and you're getting to use the latest open source models.
Hopefully you guys check it out.
This is my go to provider when I do use open source models.