It is week two, day four.
It's the start of a two day series in which we wrap up week two, and together, the two days will be
that same combination of challenging and satisfying.
As you will see, it is, of course, a yellow day.
It's a yellow day because we're doing integrations, in particular Super Bass, which you had your first
look at.
Hopefully you logged in, you clicked around, you set up your organization, done some of the initial
stuff, got some sense of what we're talking about.
But first, as usual, we have some recapping.
I want to just do a quick refresher on rag, because the next couple of days are all about building
rag ourselves, for reals.
And the first, a quick refresher on what rag is.
It's, of course, about making your model appear to be more knowledgeable.
It's retrieval augmented generation, a trick that works well.
The simple version of it is they just take some information, shove it in the prompt.
So you ask the user a question and you give the model some information that might pertain to the question.
And the clever trick that's added to this is all about the technique for figuring out what useful data
could we include in the prompt.
And using this, this thing called vector embeddings, this idea that there are kinds of llms that are
good at taking some text and coming up with a set of numbers that represents the meaning of the text.
And then you can use these numbers as a way of finding similar information.
And that gives you a way to look up stuff in your knowledge base.
That's likely to be useful in answering a question.
And at the heart of this new idea is a type of LLM that you may not have even known about, called an
embedding model, also known as an encoder, an embedding LLM.
And it's something which is able to take an input as some text.
And the output is a bunch of numbers, a list of numbers which are called vectors, also called vector
embeddings.
And these numbers represent the meaning of this input text.
and you can think of it like it's a point in space.
If it were three numbers, it would be a point, like here.
If it's a thousand numbers, then it's a point in multidimensional space.
But basically it's a bunch of numbers and text with a similar set of numbers that are close to each
other in this space, represent similar kinds of stuff, and that allows us to search for relevant contextual
information in our knowledge base.
Something that has similar meaning pertains to a similar topic as the question that the user is asking.
And so, you know, I'd love to get this diagram.
You've seen this diagram already a few times.
And if you've done my other courses, you see there's like a million times you're like, oh, not this
again.
I'm sorry, I like this diagram.
The user asks a question.
That question comes to our code or to wherever we've got our workflow.
And what we do is we say, I want to know the vector or the vector embeddings associated with this question.
If the question is how much does it cost to fly to Heathrow?
Turn that into a vector.
And the next thing you do is you say, I've got a database of all my information.
I've already turned it into lots of vectors.
And what I can do is retrieve the closest pieces of information to the vector for that question, so
that if the question is how much does it cost to go to Heathrow, then hopefully ticket prices to London
is some of the data that gets retrieved.
And when you retrieve it, you don't you don't retrieve the vectors.
You retrieve the text, the original language about the ticket prices to London.
And it's that language which you put in the prompt to the LLM.
You say, hey, LLM, the user has asked this question, how much does it cost to fly to Heathrow?
Here's some interesting relevant context.
And it includes ticket prices in London.
And the LLM is good enough to to connect the dots to get that input context.
And it knows that the most likely next tokens will be the answer to the question.
And that's what comes back.
And that is Rag.
And you know, when I give these recaps, I like to add some something extra, uh, to spice it up.
And I will just mention that often when you're building these vector databases, you've got content
that's something like lots of documents with lots of stuff on them.
And one challenging question is how do you create like one vector for the whole document?
Or if the document has, say, ticket prices to different places, maybe you should break it up.
And for each of the paragraphs of text or for each of the sections, you should make that be a little
miniature document with a vector associated with it that might get inserted into the prompt.
And this process of potentially taking a document and breaking it up into smaller pieces is called chunking.
You're turning it into chunks.
And there's again, there's a whole cottage industry of different techniques and strategies for chunking.
And people like to get very excited about it and come up with lots of sort of principles.
You should try it like this.
You shouldn't chunk like this.
Uh, the at the end of the day, they're the only one true principle is that you need to test it.
You need to try different strategies, different approaches to find out what works best for your data
and your questions.
And if you take my AI engineering courses, the core track, I go on and on about it and we build testing
methodologies, and we use things like MRI and all sorts of stuff to get to a point where we can try
out different chunking strategies and pick the one that works the best, but that's that's more of the
pro stuff.
What we're going to be doing today is just chunking using defaults, which often works very well indeed.
Okay.
And then the other thing that we covered yesterday is rag versus agentic rag, which is just the kind
of next step to rag.
So traditional rag you remember I said someone is chatting with us.
We do vector based retrieval.
We find vector and we we look it up in our knowledge base.
And then that gets sent to the LLM to respond to the question in a genetic rag.
It's just a bit more than that.
The idea is that the user asks a question to our agent.
Our agent is powered by an LLM that doesn't just respond to the question, but also it manages the whole
workflow of how to respond to the question.
And so it is equipped with a tool that allows for vector based retrieval, and it can choose to use
that tool in order to find relevant contexts.
And maybe there are some settings that can it can apply to decide how to use the tool.
And maybe it's also got other kinds of retrieval tools as well that in its in its arsenal of tools that
it can use to try to best answer the question.
And it can figure out how to pick the right ones, and it can try all of them and use that to to assess
context.
And if the context is not good enough, it can do it again.
And that kind of iterative approach is, is what makes it agentic where the LLM is deciding how to do
that.
And so that gives you a rough sense of rag versus agentic rag.
And now you often hear people saying that rag is dead.
Uh, and they usually say that for one of two reasons.
One reason is because context windows, which is the total amount of information that you can pass in
to an LLM for it to be able to respond to you loosely.
Loosely speaking, uh, context windows are getting bigger and bigger and bigger, and so you can cram
more and more into that context.
And so they get to a point when you think, hey, why do I even need rag?
I can just put my entire knowledge base in the context window.
And so we don't need rag anymore.
And now I think that is probably a bit of a red herring.
Uh, ultimately, uh, to be scalable, you would need to have some sort of rag at some point, like
we're about to build out a rag pipeline for a company, and they might have tons and tons of documents.
They might have gigabytes of documents.
You could easily get to a point where you would far surpass what could possibly go in a context window,
but also, surely it's a waste of compute and resources and and just just a general waste to put in
tons of irrelevant content into the context window.
If if the LM is being asked for ticket prices to London, there's surely no benefits to giving ticket
prices to every other city in the world in the context window as well.
So.
So it seems to me that the idea that the context window has got so big that we don't need rag anymore,
that's a red herring.
That's not right.
But there's another reason.
Some people say rag is dead and it's because of a gigantic rag.
A gigantic rag has come along and shown us this more iterative, more autonomous way to be finding retrieving
relevant content.
And so it's possible that traditional Rag is going to be fully replaced by Agentic rag.
And I think that's probably true, but I think that that's just a terminology thing.
I think Agentic rag is the natural successor to Rag, and it's the way we do rag nowadays.
I don't think that means rag is dead.
I think Agentic rag is just the natural evolution of the same kind of trickery.
And so I say, long live agentic rag.
We're in good shape.
It's here to stay.
Okay, so there are two distinct phases to building out rag.
The first of them is data ingest, pulling in the information we've got and putting it into our knowledge
base into our vector store.
So you typically have some, some data in some some source place that could be text files.
It could be a sheet, it could be PDFs.
And you need to to load in that data.
You need to what's called extract it pull it from from the source.
The next thing you need to do is what they call transform, which is saying it's in some format in the
source.
We might need it in a better format for us to work with.
Often you use this to add in metadata, little extra bits of information for the data that might be
used to filter it, or something like that.
So transform is typically what you do next.
And then uh, you would do a few things that are very specific to rag.
You would typically chunk, which is what I mentioned a few minutes ago, about breaking up your document
into pieces that feel like they're going to be about the right size.
That's likely to match the questions that are asked.
And there's so many different ways of doing this that the default way is just to split it into things
that make sense.
There's a more sophisticated way called semantic chunking.
When you try and work to break it into fragments that are most likely to be about the same stuff, but
often the simple approach works just fine, and that's what we will be doing.
See you chunk it, and then you do what's called vectorizing, which is informal.
The proper name for it is is encode or embed or calculate the vector embeddings, which is basically
putting it through an embedding model and coming up with the vector.
But the word people often use is vectorize to find that vector.
And then you load that into your vector store.
And the reason I use this terminology is that you may have heard of extract, transform and load ETL.
That's that's really like a traditional data engineering expression from many years ago.
And we still very much think in terms of extract, transform and load today.
But there is also this chunking and vectorizing that happens along the way.
So these are the data ingest pipes that you would typically build as part of your Rag solution.
And then the second distinct phase, which is sometimes called the Rag pipeline, or the question answering
is where the user asks a question.
Obviously it goes to our agent.
If this is Agent Rag, there's an LLM that manages the workflow.
And one of the things it does is it calls a tool which is able to look up relevant information.
And the way it does that is it.
First vectorize the question and then it looks that up in the vector store.
What information do I have?
What chunks do I have in there where the vectors associated with the chunks are similar to this vector.
And then I get back those chunks.
I get that text and that is what I use to then call the LLM to get my final response.
And that is the second distinct phase of building Rag.