-: In this video we're gonna cover Langchain Vector stores
and also we're gonna have a look
at Langchain's Indexing API
and what this means about when you're doing data ingestion
into your vector databases.
Firstly, you're gonna have to run
a couple of Python imports for this
and I'd recommend running this specific notebook
on a local computer rather than inside Google Colab
'cause we're gonna also run a Docker command as well.
So you'll need to install langchain langchain_openai
and langchain_elasticsearch and faiss-cpu.
Once you've installed those,
we're gonna import a couple of different things.
So a TextLoader, OpenAIEmbeddings,
a CharacterTextSplitter and Faiss,
which is another vector database.
Now if you haven't already,
you'll need to set up your specific OpenAI API key.
And we've just got some raw text here
that we're just gonna write to a file.
And then once we have that file saved locally,
we're then gonna load that in with the TextLoader
and we're then gonna split that
into chunks of 1000 at a time with no overlap.
And then we're gonna split those
into Langchain documents that you can see here
you get a list of Langchain documents
and then we're gonna load our vector database from this.
And this is just giving you another taste of the fact
that there's lots of different methods
on side of the vector database that you can use.
So if you do db.,
you'll see you can add lots of documents,
add embeddings, you can add text.
And the interesting things that are quite important are,
for example, you've got the ability to look at the index
and you can also save this,
but I specifically recommend having a look
at these similarity search.
So you'll see when we do
a similarity search of digital marketing.
At the moment, we only got one document.
And you can also see the actual,
this gives you a tuple object
and each of these has both the document
and the the relevance of that document.
Now you can also change the K as well,
so that's quite nice.
Now if we have some more documents,
you can also load documents specifically
from Langchain documents.
And you can see here we've got two documents here.
We've got the page_content of James Phoenix
and we've also got the page_content,
digital marketing is a growing industry.
Now the important thing
when you're doing Langchain indexing is to remember
to add specific bits of metadata
because it's gonna use both the page content,
so what's inside the page,
as well as also hashing your metadata dictionary
as well as hashing the page content together
to create a unique ID
that's kept inside of Record Manager.
So that's really important that you add specific metadata
to tell the Langchain indexing exactly
where these documents came from
so that when you do then start using an indexer on those,
you can effectively deduplicate the documents.
So you can see, now if we look
at the different LangChain documents,
we've also got that metadata associated
with (indistinct) documents
and we can add these documents
and Faiss will give us back a unique index ID
for these documents and we can do a similarity search.
And if I do for James,
you can see I can lock in the K at one
and we can see that the document
that you get returned is this
James Felix worked in digital marketing for three years
and also we've got that metadata as well.
Now the problem is if we don't use
indexing or a record manager,
there's no way to know necessarily
when you need to reingest your documents
whether anything's changed.
Without knowing how those documents have been indexed,
it's very difficult to keep track of specifically
what documents need to be reingested,
what vector database,
bits of the vector database need
to be pruned and deleted or updated
and this means that you want to solve this problem.
So Langchain have come up with a really nice way
of solving this called the Indexing API.
And I've linked to that inside of here.
And basically how it works is
you've got a couple of different components.
So one of the components is a record manager
and the record manager basically says,
we'll take the document,
which is a hash of both the page content and the metadata,
and then what you can do is you can then deduplicate
based on that document hash.
And so when new documents are ingested,
if the page content's changed
or the metadata has changed for a specific document,
then you need to do some different,
different types of operations to clean the vector database.
There's different types of deletion modes
and we'll cover that towards the end of the video.
For now, what I'd like you to do is
just have an example of this.
Now you will need to have Docker installed
on your local machine to run this.
And what I recommend doing is copying
this docker run command, loading up a terminal window
and then running that docker run command.
And that will ask you
if you want to install a load of packages.
And you'll see here, for my example,
we are loading Elasticsearch
and basically this is a vector database
that we can use locally.
Now what's interesting about having
an Elasticsearch installed from a Docker,
a container is then we can load that up on this port.
So it's localhost:9200,
we've got an index name
and we've got the embeddings
coming from open AI embeddings, okay?
Now you also are gonna create something called
the SQLRecordManager.
And remember the SQLRecordManager really does
a couple of different things.
What it's mainly doing is it's mainly storing
what documents have been indexed, what is the document hash,
and then it allows you to keep track of specific records.
So I can run this record_manager.create_schema,
and that creates the database schema.
And then what's interesting about this is
we're just using SQLite at the moment,
but you could use any SQL database you want
for storing the specific types of records.
Now let's see what happens when we pass in a document
that's got exactly the same kind of metadata,
but it has different page content.
So you can see if we scroll up,
the page_content was James phoenix worked
in digital marketing for three years,
but now we're saying that James phoenix worked
in digital marketing for seven years.
So we've updated one document,
also you'll see here we've got this little helper function
which allows us to clear content.
So we're gonna go and clear the index to start with.
Then we're gonna go
and with the documents we've already created,
which are just above here, you've got these documents here
and what we're gonna do is we're gonna add those
specifically to the index.
And what you'll see here is
because our record manager was cleared,
basically we've got two documents.
And so for the first time we're saying
use the source_id_key.
So this is the key that we're duping against,
which is the original source.
And we're basically saying
with my record manager is now gonna say
for this rector store, insert these documents.
Now what's interesting about this is
if we have these documents here
and you can see I've got these two documents,
if we then were to rerun this exact same command
and because we are using this cleanup incremental,
so because these documents haven't inherently changed,
there's nothing that's gonna happen here.
So if when you see when I rerun this,
it skips adding any documents
because the document hash hasn't changed.
Now, if I was to then update one of these documents,
so if you see here, we've got these updated_docs
where we've said James phoenix has worked
in digital marketing for seven years,
when we try and update a single document,
we actually end up deleting one of those documents
based on the source and then adding a new one
based on the fact that document has changed
and we've ingested it into the vector database.
You can also see what it would be like
to add a new document.
So if we just had a new document
within new different types of metadata,
you'll see that we've got a null added,
but we haven't necessarily got any that have been updated,
skipped or deleted.
And also if we already have something that's the same,
then essentially what's gonna end up happening is
you can see we're skipping the document.
So from the document that we just tried to insert
because nothing has changed,
we've getting a skipped document.
Now there's three modes
that you can operate the Index API in.
We've been primarily using the one that I'd recommend,
which is incremental.
And so what happens with incremental is
it deduplicates content as you,
for each individual document as you reingest it.
Now if you do Full,
it will do that at the end of the indexing,
and if you do a Cleanup Mode of None,
then basically it's up to you
to clean your specific vector database.
So again, the key points of this video
and for you to take away is always make sure
that you add on metadata to your Langchain documents.
You can have a record manager
and that will be a database that you can see here.
I've got this record_manager_cache.sql,
but you could have this specifically
in any kind of SQL database that you want,
whether you're using Postgres
or whether you're using my SQL.
Essentially all you need to do is change this db_url
and add on the various connection strings
with the namespace.
And then that's an important point.
The other important point is remember to pick the mode
that you're most interested in.
So I would recommend using incremental.
And the reason why all of this is important is
because this means that you can have a reingestion pipeline
that's powered by a record_manager and a vectorstore,
and then it's being deduped
by this source_id_key plus a document hash,
which means that when you rerun your ingestion,
it doesn't have to always reingest every single document
and you can have smarter ingestion pipelines
into side of your vector database.
Cool, I'll see you in the next one.