So let's first start talking about what it is LM ops.
So LM ops refers to the practices and tools used to operate and maintain LM applications in production
environment.
This terme is analogous to DevOps or ML ops, but specifically adapted to the peculiarities of LM applications.
So DevOps is the terme we use in conventional software development, and ML ops is the terme we use
to, uh, apply to the machine learning traditional, uh, environment.
LM ops covers the entire life cycle of an LM application, from the development to deployment monitoring,
uh, to their continuous updating and maintenance.
Sometimes we get confused with terms like observability, monitoring, evaluation, guardrails, and
others regarding LM ops.
So let's talk a little bit about these four topics.
Lopes represents a comprehensive approach to managing LM applications, encompassing everything from
development to maintenance of these apps in production environments.
Unlike more specific concepts such as observability, monitoring, evaluation, and guardrails, which
are components or aspects of LM ops, this terms encompasses the complete operational management of
LM ops.
These other components are critical to the success and sustainability of lmps in the real world, ensuring
their performance, reliability, and ethical compliance.
So let's talk a little bit about.
Each of them first.
Observability.
It refers to the ability to understand the internal state of an LM application from its external outputs.
So when you say internal states external outputs, it's talking about questions and answers.
So observability is going to focus on the quality of an LM application A to get good response a accurate
accurate responses from questions.
Observability is an observability in an Elm application implies having visibility on how the model processes
and responds to inputs, how the Elm application processes and responds to inputs, which is crucial
for diagnosing problems or understanding its behavior.
So observability is one component of Elm Ops.
Okay.
Second component that we are going to know a little bit better.
Monitoring.
Monitoring is another component of LM ops focused on the continuous surveillance of the performance
and health of the LM application in production.
This includes tracking key metrics such as response latency and set accuracy and resource consumption.
Monitoring is essential to ensure that the LM application functions as intended and to quickly identify
any problems.
Okay, so observability focus on questions and answers.
Monitoring questions on key metrics A monitoring focus on key metrics.
Evaluation.
Evaluation involves the periodic assessment of the effectiveness and accuracy of the LM application.
Evaluation can include testing with new data, comparisons with benchmarks or standards, and analysis
of user feedback.
It is a crucial step to ensure that the LLM application remains relevant and useful over time.
So evaluation when you are a in the last stages of development of your professional LLM application
evaluation means testing and testing is different in LLM applications from a conventional software applications.
Remember the expression lack of reproducibility that we have used before.
This is uh applied here.
So in the same question is not always going to have the same answer in an LLM application.
Even when the meaning is the same the wording may differ.
Okay.
So a conventional software application will always give you, if it is correct, will always give you
the same answer to one, uh, question in LLM applications.
If I, for example, ask a chatbot, the wife of Napoleon, in some cases, uh, is going to give me
just the first name in some others are give me the first and the second name.
Uh, in some other cases, the same chat, uh, bot is going to tell me, you know, the, the aristocratic
title of the person or whatever.
So the response is going to be accurate or correct, but we cannot measure this accuracy the same way
we, uh, we use with a conventional, uh, software applications, because we will with a conventional
software application, we will say every time the user asks you two plus two, the answer is going to
be four.
And that's it is nothing else is four.
If it is, four is correct, the test is correct.
If it is not for the test is not correct.
With early applications, there's a different scenario as you as you may see.
So we need different approaches and different tools.
We will talk more about that later.
So evaluation is going to be an LM ops, uh, component in the final stages of development of a professional
LM applications, and also during the life of the application.
Last thing and that is is worthy to to to consider now guard rails.
So guardrails are control and safety measures implemented to ensure that the behavior of the LM application
remains within acceptable limits.
They include restrictions on the application responses, filters for inappropriate content, and safeguards
against biased or misused.
They are an essential part of risk management in LM ops.
So if you remember a immediately after the launch of ChatGPT, we started reading news about, you know,
how easy it was to learn how to make a bomb or, you know, any other criminal activities, learn any
other criminal activities using ChatGPT.
So.
This, of course moved and the company behind ChatGPT to start using guardrails in order to prevent
all these, uh, improper, uh, data or information or content.
Right.
So right now, if you go to ChatGPT and you try to ask about these kind of things, uh, 99% of the
times, the chat GPT, uh, application is going to respond with a negative, like, I cannot answer
this, I am not authorized, blah, blah, blah, or this is, um, whatever the whatever the answer,
it will, uh, respond to a guardrail policy that, uh, ChatGPT has in place.
Right.
So this is, uh, one thing that you are going to see more and more in LM ops and one thing that, uh,
it was interesting to talk about in this stage.
So, as you know, we are not going to talk about LM ops in detail.
What we are going to do is just to tell you briefly the kind of things you will like to study further
once you are in this, uh, in this scenario of preparing the launch of a professional LM application.
So we are going to talk a little bit more about some of these things associated with the LM ops world.
In the next lesson, we are going to talk about what it is misaligned behavior.