Eden: Hey there, Eden here,
and in this video I wanna talk about
integrating agent into a production environment.
And specifically I want to discuss
the challenges in doing so.
(graphics whooshing)
So, we saw that when working with agents,
we are using a lot of LLM calls
because we're using the LLM as a reasoning engine.
So, every step we're going to make
and every tool we're going to use,
it needs to come after an LLM call
where the LLM has decided to use that tool.
And this will result in multiple calls to the LLM,
but not only will have multiple calls,
but those will be sequential calls, one after another,
where each one is waiting for the result of the prior one.
So, this depends how complex is our task
and how many reasoning steps do we need to make,
and can easily escalate
into a very long running application.
So, we need to take that in mind.
And there are some workarounds for this,
like using a semantic cache and using an LLM cache,
but we'll not discuss it in this course.
The next thing I want to talk about is the context window.
You notice that we send a huge prompt to the LLM
every time we make a reasoning step.
Now, most of LMS nowadays
can handle around 32k tokens.
It may sound a lot,
but in a real-world application,
we can easily surpass that limit.
So, eventually, we are limited
by the number of steps we can make
because of the context window.
Now, I know there are models like Cloud Anthropic,
which can receive up to 1k tokens,
but de facto working with 100k tokens
sending to the LLM introduce us with a lot of problems,
like the LLM tend to forget what's in the middle.
See the paper of Lost in the Middle.
Okay, so the third thing
I want to talk about is hallucinations.
Now, we know hallucination
is when it's sent to the LLM a question
and the LLM responds
with something not relevant for that question
because eventually the LLM
is guessing one token after another.
And retrieval augmentation, by the way,
is a nice technique to reduce hallucinations
because we ground the LLM with information
that we send in the context.
Anyways, 'cause the large language model
is a statistical creature,
then this means we have a probability
of getting the correct answer.
And the correct answer
is actually choosing the correct tool to use.
So, let's assume that we have 0.9 probability
of getting the correct response
that chooses the correct tool.
Now, if we do this only once,
that's okay and that's a very good probability.
However, if we're making sequential calls
one after another and after another,
then by the multiplication law, after a few steps,
and in this case after six steps,
then we get that the probability
for getting a good answer drops to 0.59%,
and this is only for six consecutive calls.
What if we have a huge task that requires more?
Then this number drops even more.
So, there are ways to solve it,
and one of the ways is to use fine tuning.
And by that I mean to take an LLM
and fine tune it for tool selection.
So, there are research papers about that
and that people manage to fine tune LLMs
to make the tool selection, for example, to run API calls
to have them yield better results.
So, instead of a 90% chance,
the chance of getting the correct tool is much, much higher
because the LLM is fine tuned on the tools
that the agent have to its disposal.
Okay, let's talk about pricing.
Now, you know that we pay for the tokens
that we send and receive from the LLM.
Now, if you're using agents,
you saw how big the prompts can get.
And when we do this in scale by the millions,
then you can figure out that the billing report
we're going to get is going to be very high.
And I didn't even mention GPT-4,
which has a strong reasoning capability,
but runs very, very slow and is very expensive.
So, if we use that and we use it in scale,
then we can get into a situation
where it's not financially worth for us to run those agents.
So, there are a couple of strategies to solve this,
the first one is to use some kind of cache, semantic cache,
instead of making an LLM call, we discussed this.
And another one is to use retrieval augmentation
for the tool selection.
We're not going to show this in this course,
but retrieval augmentation for the tool selection
can also handle in a case
where we have too many tools to choose.
So, in that way, before we make the LLM call
for the reasoning process,
we do a semantic search and retrieve those relevant tools
that have a high probability
to be the correct tools to our answer.
So, now we need to talk about response validation.
By the way, all the topics I discussed
are relevant for all LLM application
and are real-world challenges.
So, it's not only for agents,
it's also for every LLM application.
So, because we're making a call to the LLM
and we're counting on it
and based on that response
we're going to maybe choose some tool
or maybe output something to the user,
then we need to have a mechanism that validates this.
Because even if the LLM responds us with the correct answer,
but it's not in the correct format,
then it can mess up our application.
So, testing it is a very complicated task.
And I personally haven't encountered
a robust solution for this issue.
Okay, let's talk about security.
And in iGenetic applications,
we give the LLM capabilities to do stuff.
For example, to run queries against the database,
make an API call, or talk to a third party,
and the agent has permissions to do all of this stuff.
So, if some malicious user hijacks our prompt
with prompt injection
or get a hold of our API key,
then they can have access into our tools.
And if our database is proprietary
and we don't want to expose it, then we are in a problem.
So, security is a big issue,
and overall we want to adhere
to the least privilege principle,
and that is to give those tools and to those agents
the minimum permissions as they require.
We want to have guardrails on the prompts
we send to our agents
and to allow and not allow maybe some prompts.
So, there are many open source solutions for this.
Now, I recommend using LLM Guard,
which is an open source
that I think is a very promising one,
which offers a lot of functionality
when it comes to LLM security,
integrating LLMs in production.
Now, the last topic I want to talk about
is overkilling it when using agents.
Agents are good when we have a deterministic sequence
of steps we want to execute.
If we know exactly what we want to execute
and we can define it by talking or by writing code,
then we don't need to use agents.
I've encountered multiple people in companies
trying to achieve things with LLM agents
where the real solution and robust solution
was to simply implement some deterministic code in Python
that would achieve what they wanted.
So, my advice to you before you use agents,
really think if you can implement it yourself
in some deterministic code.
If you can, then I really don't suggest you using agents
because, as you saw, it has a lot of challenges.
And that's it for this video.
And I really want to give a quick disclaim over here.
I think agents are an amazing technology
and they have a massive potential for us
to achieve great things.
I don't think it's easy going from a prototype
to a production application.
It is totally possible, but it has a lot of challenges.
By no means, I'm not saying
that agents are not ready for production, I didn't say that.
I just mean that we need to be very careful
when we use them because it comes with a lot of cost.
And integrating such great technology
comes with a cost, and we saw the challenges.