Hey there.
Eden here.
And in this video, I'm going to cover some of the important concerns when dealing with managed large
language models.
And I want to cover here some key points about data retention and privacy.
And I think those are very important issues that we should be aware of at least.
And when taking a generative AI application to production, which uses large language models, then
you have to answer questions like is the data being used for training purposes, or how long the data
is being retained for and for which purposes, copyrights issues, and how can you use the generated
text, and a lot of more things that you need to be aware of and you need to answer.
Now, this is a huge, huge topic and I can go for hours to talk about it.
And there are tons of requirements and rules and regulations, etc. in this video I'm going to go only
on a few of them, so this is definitely not the full list of things you should be aware of, but just
the beginning and something to help you get started and to give you an introduction to it.
And before we begin, just a very important disclaimer, and this disclaimer is super important.
I am not a lawyer.
This is not a legal advice, and you should consult with your legal team and privacy team before integrating
any LLM based solution in your enterprise.
There are a lot of rules and a lot of regulations that I am not aware of, and the topic of data retention
and privacy is very, very sensitive and should have appropriate handling.
I am also not representing any LLM vendor here, and I'm not giving legal advice.
And every LLM vendor is going to have a Eula, an end user license agreement where they have their terms
of services and they specify how they handle your data.
And it's a legal document that you should look into.
Okay.
I'm just giving you my $0.02 here.
And again, I'm not a lawyer and this is not a legal advice.
So this is a very important disclaimer.
You should always talk to your legal team and privacy team and to act according to what they say.
This video is for educational purposes, and I'm going to give you my $0.02 on this topic.
So you should take everything I say in this video with a grain of salt.
And you should really do your own research when it comes to this topic.
Let's start by talking about data retention and what happens to our data that we send to the model to
a managed model like OpenAI, GPT four or Mini, or Google Cloud's Vertex AI Gemini.
And I want to make a very clear distinction over here.
I'm not talking about the B2C direct to consumer products like ChatGPT or Gemini, which was formerly
known as Bard, but I'm talking about the cloud APIs that they expose for enterprises and businesses.
And again, I want to reiterate, I'm not a lawyer.
This is not a legal advice.
And you should do your own research.
And if you're working for an enterprise, you should consult with your legal team.
All managed vendors have a Eula and users license agreement, and you can see all of the details there.
And this is a legal document which you can read and it has all the information.
But I'm going to give you my $0.02 in this topic.
And a lot of people are concerned what happens with their data.
And they're worried that the managed data vendor are going to train their next model on data that they
will be sending to the model or the generated output.
And from what I saw, in most cases, at least in the top tier models, then there is a guarantee that
they're not going to use the data that we send or the generated text to training purposes.
So we get a guarantee that they're not going to use their data for the model training.
At least that's the default behavior.
If we do want to allow them to do this, we can opt in voluntarily for that.
And by the way, this is a very valid concern because if we're working for an enterprise and we want
to integrate an LLM based solution, we need to make sure this is the case, because we might be dealing
with proprietary data of our organization that we do not want to expose, and we do not want to get
leaked or to get trained on, God forbid.
And if we have customers, we of course, do not want their data to be trained on, because probably
we're going to have legal obligations that is going to protect their data.
So this is the first thing I want to address.
All right.
So this was about training on data that we send to the LLM.
Of course you should expect differences between different LLM vendors.
And let's talk about data retention.
And our vendors going to save the data that we send the LLM.
And if so for how long.
What kind of rules do they have for which purposes, etc..
And in this example of OpenAI, we can see that it clearly says that in order to identify abuse, they
may retain our requests for 30 days and after it it will be deleted.
Or if there are other law requirements.
And they do mention that for some customers they may have a zero retention policy, which is available
where none of the data will be logged or persisted, and it's going to be only used for serving purposes.
And of course, this may also change between vendors.
I did see other vendors where they have a zero retention policy right from the get go, and in order
to log and to retain some of it, you would need to explicitly opt in.
So again, there are differences between the vendors and those may change over time.
All of the rules.
And one last disclaimer.
So even when providers are going to guarantee that they're not going to train their model with our data
and we're going to have zero retention policies.
And for some organizations, this is simply not enough.
Let's take the banking institutions, for example, or insurance companies, they usually have very
strict regulations and very strict policies when it comes to privacy and when it comes to data retention
and sharing customers data, because it is very, very sensitive.
And for those companies, those promises and those guarantees from the LM vendors is not enough.
And usually if they do want to integrate generative AI into their applications, they are going to self
deploy open source models.
And with self deploying open source models in their environment, they will have control over their
data, their retention policies, and they have all the control they need.
However, it will come with a cost because serving LMS is not that simple and they would need to handle
scalability, durability, availability, all those ilities, which is a lot of operations work and
it's not simple to do correctly.
It also costs a lot of money and a lot of effort because they would need to host it on GPUs, and they
would need people to maintain it and to.
Handle the deployment.
And of course they would need to handle the security.
Because even open source models can have vulnerabilities in them, which I.
Didn't discuss at all.
And it comes with a price as well.
And there is also.
This middle ground of hosting the LMS, the open source LMS within their cloud environments.
So they're going to use the managed services of cloud providers to.
Deploy those open source LMS.
So they shift the burden of all those edits to.
The cloud provider, they still have control because it's going to be their.
Cloud environment.
And they can enforce their security controls in.
Their cloud environment.
And just to conclude in this video, I just wanted to.
Introduce you with a couple of things that you need to consider when it comes to privacy and data retention
when using large language models.
This is a very deep topic and we can talk about it for hours.
And my goal here in this video is just to give you an introduction and some things to concern.