-: And in this lesson,
we're gonna cover a concept called routing.
Routing is a workflow pattern that you can use of LLMs
to dictate when a user input hits a various bit
of your system, where that user query should go next.
It's almost like a classification happening,
and then you're basically assigning it
to a different task downstream.
There are a couple of different use cases,
but generally it's when you have different types
of customer service or inquiries,
different types of LMS or model configurations to deal
with different types of tasks.
And also evaluating whether a request meets certain
guidelines or triggers.
So the first thing we're gonna do is install pip.
We're going to install OpenAI and Pydantic,
and we're gonna do a couple of imports.
And then we're also going to allow you
to put your OpenAI key,
and we've set up a client with a model of GPT-4o
Okay. So the next one we need
to do is we're gonna get a couple
of different endpoints that we're gonna hit.
The first is a function.
So we're gonna create a function called classify_as_spam,
and we're gonna take in a user_query, which is a string.
And what we're gonna then do is return a boolean.
For now, I'm just gonna put this as a pass.
We're also gonna have a generate_summary function,
which is also gonna take in a user_query,
but instead of return a boolean,
we're going to return a string
and we're just gonna put pass.
So let's build up the classify_as_spam function first.
So the the prompt that we're gonna use is a spam_prompt,
and we're gonna say, "You are a spam classifier
that takes a user query
and returns True if the user query is spam
and false otherwise."
Cool. So we've got our prompt.
Then what we're gonna do is we're gonna make our
spam_response,
and we're gonna use the client.beta.chat.completions.pass
and remember, we're gonna use that response_format
and we're gonna use a response_format of a bool.
And then as well as that,
we're also going to put the model=MODEL
so we're gonna use the messages,
and then we're gonna have a role of user
with the content of that prompt.
We're also going to turn down the tokens.
We're also going to turn down the temperature
and set that to zero to make it more deterministic.
And then we also need to add in the user_query.
"Here is the user query." So we'll go back up to the prompt.
Here is the user query and remember to make sure
that is an string as well.
We're also gonna add in some validation here.
So we'll say, if not spam.choices.message.content,
or not spam_responses.choices[0]
What I want to do is .message.parsed.
There we go.
And then we're gonna do our generate_summary function.
So we'll have a summary_prompt.
We'll also use the chat completions,
but in this case, we're just gonna return the content.
We're also going to define a router.
This router's gonna define all the different endpoints
which the function can go into.
So we're gonna have a destination.
We're gonna set the Literal,
which basically means this string either has
to be a type of spam.
And we're gonna change these actually to the real ones above
to classify_as_spam and generate_summary.
So this router has to have a destination,
is either a key of destination
with a value of classify_as_spam
or a key of destination with a value of generate_summary.
We're then gonna make a function called router,
which takes in a user_query and then will return a router.
So what we're then gonna do is say you have a router_prompt.
You are a router. Here is the user query.
And then we're going to basically say,
we then have a response format of a router,
which is then going to specifically give us a router.
So this would give us a router
and tell us where we need to go.
And then what we're then gonna do is we're then gonna use
this router to actually do that type of routing.
So we're gonna have try and accept,
and then we're gonna basically, for now,
we're gonna put a print of a router
and let's put something in that would be useful
to actually do something with.
So we'll say, "I want to make a summary
of the following text:
This is awesome,
how are you doing?"
And then what we'll see is we'll get back a router object
and notice how it decided to send this
to the destination key with a string letter
of generate_summary.
So we could store this
as the destination_router_result
and then what we could then do is print this out.
So you can now see that we could go
to the .destination of this.
And we've got the generate_summary.
So we could extend this
to then basically get the router choice.
And then we could basically do something like this
if the generate
at the destination_router.destination is equal
to generate_summary,
generate_summary, and then we have the user_query.
So let's go and avoid hard coding this user_query.
So we'll move this user_query out
from the router function up here,
and then we'll put that back here.
And then if it's not, then it's a classifier spam.
And then we'll put else raise ValueError.
So now what's happening is we have a user_query,
it goes into the router function.
And remember all this router function is doing is having a
router prompt, which is determining
what thing is gonna happen next.
And then we use the client.beta.chat.completion.parsed
with that router prompt to decide,
"Hey, given this user query,
where are we gonna send that user query?"
It will then return back the router here
with the string literals of where
that specific thing can go to.
And then as well as that,
then what we also do is once we have that destination,
then we can specifically route the user_query
to a specific destination.
This is a really useful pattern,
and it's similar to function calling,
but we are basically just redirecting the user query.
We're not necessarily asking the LLM
to generate functional arguments or keyword arguments.
We're simply doing a navigational step in determining
what should happen next.
And we have quite a lot of explicit control
over this routing pattern.
Alright, so in the next one we'll have a look
at parallelization and how we can paralyze tasks with LLMs.