What are document loaders and why it is important for you to understand a.
The versatility of them, the number of things you can do with them.
So the important thing of this chapter is to understand that with launching, you can extract a data
from a lot of, uh, sources and formats so you can extract data from web pages, databases, YouTube,
videos, Twitter, Excel, pandas, notion, Figma, Huggingface, GitHub, whatever you want, and
in many different formats.
You can extract data from PDF, from HTML, from Json files, from word files, PowerPoint, etc.,
etc. so that's the most important concept to get.
From this chapter we are going to experiment with three examples.
We are going to load a data from a PDF document, extract data from a PDF document.
We are going to extract data from a YouTube video, and we are going to extract data from a web page.
The first exercise is going to be very simple, very easy and very fast.
The second one is a little bit, uh, slow because getting extracting information from a YouTube video
can be, uh, can can take time especially.
Well, depending on the length of the, of the video.
And you will see that in this case you need to install a module called ffmpeg.
Right.
This is this is very easy.
Remember if you have any problem installing these packages you can go to ChatGPT and ask how to do it.
Or go to Google or go to StackOverflow.
Remember all these uh, usual places where we find information on a daily basis so that, uh, regarding
the, uh, YouTube data extraction.
And finally we will, uh, we will have three exercises, uh, about loading content or extracting data
from a web page.
So we will see three different options.
And in this case we will see the necessity to clean and prepare the loaded data before using it.
So this is the one of the main tasks of the data scientist.
Uh, clean data.
Because usually when you extract data from, from whatever source you, you, you want, usually the
data is not going to come in the exact format you want it to be.
So after extracting the data, you will need a preparation phase, right?
You will need to clean the data and to prepare it according to whatever you want.
So you will see in this exercise that sometimes the data especially you will see it here.
The data coming from a web page is probably, uh, in need of a cleaning and preparation, etc. so let's
go to the code in the right side of the screen.
Uh, first, as always, we connect with the dot m file where we have the credentials in order to start
using, uh, OpenAI, uh, the ChatGPT, etc., etc..
Right.
So the first thing we do is we extract data from a PDF document.
You will need to install Pi PDF.
You know, you can do it from the terminal or and also from the Jupyter notebook.
And once you have that you can start, uh, you know, a creating the and the, the extraction and performing
the data extraction, you see with just two steps.
And once we have the the data, the extracted data in this variable, we can start asking questions
to this variable, like, okay, how many pages do you have a what is the content.
The 500 initial characters of the of the first page, etc., etc. as you can see, this is just a user
guide and not not important about, uh, how to extract data from a YouTube video.
So we are going to extract the data from the YouTube audio.
Right?
And we are going to convert this audio into text in order to do that.
As I was saying, you are going to need this package installed in your computer.
If you have a mac.
It's very easy to do with homebrew or with with brew.
Uh, but in case of that doubt, you go to ChatGPT and ask how to do that, and then you load, you
know, the necessary modules.
You install the necessary, uh, packages to, to proceed.
And then the process is very easy.
It's not super fast, but is very easy.
In this case we are extracting the data.
From a not very long video.
I think it's like seven minutes video, something like that.
And as you can see, we are extracting the well.
We are extracting the audio and converting it into text.
So it's a very, uh, interesting, uh, interesting tool.
Okay.
Third extraction websites.
We are going to use three options.
The first one is with the web based loader, uh module okay.
You will see the process is very easy.
But when you look at how we get the data, you see it's not it's not super nice.
Right.
So you are we we will need to clean and prepare this data further.
The second option is with the unstructured HTML loader module.
So in this case our data is in a more compact, uh way.
But also we require preparation and cleaning.
And in the third option with beautiful soup you can see that it's like a solution in the middle of the
previous two.
So it also requires a preparation.
So the more the main point of this chapter is to understand that with Lang Chain, we have the possibility
to extract data from many different sources and many different formats.
Okay, so this is the important thing.
The second important thing is to remember that in the Lang chain documentation, you have information
about all the data loaders and about all the operations you can perform with them.
That's the that's the place to go whenever you want to look for a data loader that, uh, responds to
the particular necessity you may have for a LM application you are building.