Eden Marco: Hey there, Eden here.
And in this video,
I want to address a commonly asked question
about using LLMs in production.
So, this is the debate whether to use an open source LLM,
like Deepseek, like Llama 3.2,
or to use managed large language models
like GPT-4o mini by a OpenAI,
or using Sonnet by Anthropic,
or using Google Gemini.
And my point of view is going to be the point of view
of enterprise organizations
and what do I suggest them doing?
Of course, there is no one size fits all,
and I think every use case
should be considered independently.
And I'm going to give you my general opinion
and I will try to touch on as many aspects
as I can when it comes to this debate.
Just a very important disclaimer,
and this disclaimer is super important.
I'm not a lawyer.
This is not a legal advice,
and you should consult with your legal team
and privacy team before integrating any LLM-based solution
in your enterprise.
There are a lot of rules
and a lot of regulations that I'm not aware of,
and the topic of data retention
and privacy is very, very sensitive
and should have appropriate handling.
I'm also not representing any LLM vendor here
and I'm not giving legal advice
and every LLM vendor is going to have a EULA,
an end user license agreement
where they have their terms of services
and they specify how they handle your data.
And it's a legal document that you should look into, okay?
I'm just giving you my 2 cents here.
And again, I'm not a lawyer and this is not a legal advice.
So, this is a very important disclaimer.
You should always talk to your legal team and privacy team
and to act according to what they say.
This video is for educational purposes,
and I'm going to give you my 2 cents on this topic.
So, you should take everything I say in this video
with a grain of salt
and you should really do your own research
when it comes to this topic.
All right, so let me start by saying that right now,
at least in 2025,
open source models have gone a long way
and they are becoming quite good.
And we have models like Deepseek,
which has amazing results in benchmarks
and can actually outperform managed models.
And in the future, it is inevitable
that we're going to have more and more open source models
which outperform those managed large language models.
Okay, so let's start
about using open source models in production.
So, first of all, allegedly it's going to be cost effective
because they're free,
and they're cheaper sometimes to operate
because they're smaller,
they proprietary models.
So, this is going to be with a question mark
because I don't believe in large scale
that they're going to be actually much cheaper.
All right.
And let's talk about customizations
because organizations can fine-tune those models
for specific tasks or domains
that potentially can outperform
those general purpose proprietary large language models.
And I think one of the major advantages
of using open source models in production for organizations
is the control and privacy.
So, companies can host their models, their sales,
on their internal servers,
so the data doesn't leave their servers
and they can keep it safe and private.
And I think this is especially true
for companies which are highly regulated,
like banks, maybe hospitals
or organizations dealing with health data
which are under compliance and under heavy regulation.
All right, so let me now tackle those advantages.
So, cost effectiveness is actually not true in my opinion
because this means even though the model is free,
because it's open source and we can actually use it
or locally in our machine,
but deploying it so that it can actually be served
to customers in large scale is a very, very hard task.
We need to handle availability, durability,
scalability, all the agility.
We need to handle security
and we need to handle a lot of things.
And now our task becomes,
rather than developing an LLM application,
but making an LLM model be able to serve our customers
and to handle our scale.
So, I think this really derails the goal
of using open source models
because it shifts all the responsibility
and all the work to start handling a lot of operations.
And one might argue that we can use a managed services
that host those open source models,
so to use services like Grok.
So, they are correct, but once we do that,
we actually lose a lot of the benefits
of using open source models
because we wanted them to be on our service
so they can be private
and we'll have control over what's going to be generated.
And once we use those services like Grok, then we lose that.
And let's talk about pricing.
Whether we deploy it ourself, it's going to cost a lot.
If it's not going to cost a lot of compute
and GPUs to serve those models,
it's going to cost a lot because we need
to pay for engineers to deploy them
and for operations team to handle them and to monitor them.
So, a lot of costs is involved of deploying those LLMs,
open source of LLMs.
And if we'll go even to managed services like Grok,
to use them,
then the pricing is not that compelling
and it's not that cheaper from those managed LLMs,
proprietary LLMS from vendors like OpenAI,
Anthropic, and Google and many other more.
And the trend of those first party models
are going to be that they're getting better,
faster, and cheaper as we go in time.
All right, so let me talk about benefits
of using the managed LLMs.
So, those proprietary models that are offered as a service.
And let's talk about the advantages.
So, first of all, the ease of use.
They're super easy to use
and they're simple to integrate
and we don't need to handle deployment.
So, it reduced the time to market
because we can simply plug and play.
They're very reliable and offer support
because the vendors provide professional support and updates
and optimizations as we go in time.
And regarding compliance.
So, most LLMs are actually compliant for SOC 2
and for HIPAA,
and every vendor can tell you if they're compliant or not
and to which compliance issues.
All right, and let's talk now also about performance.
So, those,
managed LLMs of the big vendors
are usually quite good and they offer very quality results.
And let's talk about the elephant in the room.
So, sending a sensitive data to a third party service,
it may not be suitable for all organizations,
however many organizations are cloud-based.
And if they're cloud-based,
then their data is already in the cloud.
So, they're storing their databases in AWS
or in Google Cloud, and the data is already there.
So, why is it so scary sending some prompts
to those vendors?
Because for example, let's say Anthropic.
So, Anthropic, has their model available
also on AWS Bedrock and on Google Cloud.
So, those customers which deploy their services
on those clouds can actually consume the service.
So, it's going to stay on the cloud already.
And if you're example using Gemini in Google
and you're already deployed on Google Cloud,
then it doesn't differ than using another database
or another managed services.
All right.
And I also want to address a fine-tuning.
So, you can actually fine-tune proprietary models
and the vendors actually offer this capability.
However, I'm not personally a big fan of fine-tuning
because in my opinion, in most cases,
we don't really need it and it's simply going to cost us
whether it's time creating our dataset for fine-tuning
or the actual compute to fine-tune the model.
And I think because models are so much better now,
and with the right prompt
and right few shot examples,
we can achieve awesome results which are acceptable
and won't require us to go and fine-tune a model.
Let me know if you want me
to make a video elaborating on this topic.