-: Alright, welcome back.
So you've been able to now summarize documents,
split documents, and working with different types
of summarization techniques.
Let's step it up an notch
and assume that you've got a range of documents
that you want to specifically tag with certain features.
This is a process called tagging.
Let's have a look at an implementation inside of LangChain.
Firstly, we're gonna import a couple of different packages
such as LangChain, Pandas and Nest_asyncio.
After that, we're gonna use a different document
loaded to the one that we used previously.
This one is called a sitemap loader and it allows us
to automatically load webpage from a website site map.
We'll use the standard chat model from OpenAI
and we'll also use something called
this create tagging chain function.
Firstly, let's have a look at the sitemap loader.
We've loaded in this website sitemap
and we've set the request per second to five.
After running sitemap loaded dot load, we'll actually end up
with a list of LangChain documents inside of this variable.
Next, what we need to do is define the schema
that we want to look for and tag
on each individual document.
You'll see when we have a look at these schemas,
it is a JSON schema and we've defined
three individual properties with their various types.
It's also possible to add descriptions.
So here we've added a description for the primary topic
and we've also made sure that every property is required.
We also use a different flavored model of GPT-3.5 Turbo,
which allows us to do open AI functions.
We then use our create tagging chain function,
injecting in the schema the LLM, and the output key.
After that, then what we're doing
is setting up a Python list
looping through every document, printing the documents
that you can see here with the page content
and we're invoking and calling our chain
the input being equal to the doc dot page content.
So each of these bits of page content gets injected
and inserted into the chain.
The chain's output,
which we've defined here as you can see on the output key,
where they grabbing that and storing that
inside of this Python list.
Let's have a look at the Python list then.
So in the Python list, we're now getting back some
structured data against each individual webpage.
Perfect.
Now we can also import that same data
into a Pandas data frame,
and you'll see that if we look at
a specific LangChain document,
we can actually access the dot metadata property.
This metadata is a dictionary and
because we use one of LangChain's loaders,
it automatically added some metadata for us.
So you'll see that we can use things like the source,
the location, when the webpage was last modified,
and some other site map properties.
In our case, we just want it to assume
that we are really interested
in just adding back the URL
to each of these bits of structured data
so we can loop through for, in this case,
the first 10 URLs, getting the source of that metadata,
and assigning that to the URL.
And in this instance, you can now see
that we've got the sentiment aggressiveness,
the primary topic, and the URL.
So we've been able to easily tag individual documents
with unique features inside LangChain.