Okay, so in this section we're going to build out a parser robots parser from scratch on a real site,
not just the usage example that we had from the GitHub, but we're going to build it up using either
something like Facebook, getting the robots or text from Facebook or Craigslist or another site.
I'll find out what we're going to use in the end.
So let's first import the robot parser.
So we say const require robots parser.
And in here, let's say that we have to also have a some some requests going on because we need to we
need to use request to download the robots.txt file.
So we need to edit the package of request promise.
So we say yarn request.
Request promise.
And let's see.
So we say request require.
Request promise.
And now what else do we need?
Then we need to find a URL to download the robots.txt file from.
So for this example.
I decided to use a site that's maybe not too popular for some people.
Um, it's called Textfiles.com.
So.
Okay, this is just a brief intro about the the site, even though it doesn't matter what side it is
we're scraping, but it's just a site I had a lot of fun with in my younger days looking at old files
from the old BBS in the 1980 to 1995.
It's pretty funny files to read about that people were writing back in the old Usenet BBS days.
So yeah, as you can see, it's just lots of text files that you have at the end.
But anyway, it's not so much about the site, it is more about the robots.txt file and the text file
that Textfiles.com does not have a robots.txt file.
Actually you can see they don't have one.
But the mirror, I believe the mirror on the German mirror here.
Text files Dot Müller or however you say it dot net.
So if I go on to robots.txt here, I get lots of rules that I can use.
I also had other pages in mind such as Wikipedia, but I'm just worried that some of you might make
mistakes and in the end you might get banned.
Actually, um.
So that's not that cool.
Also, some of you guys have been blocked on Craigslist or Craigslist.
So that's why in the end, I decided just to go with text files.com or this one.
The mirror text files merely or however you say it, I'm going to put a link for it so you guys can
just use it.
And.
Okay.
Anyway, enough about the story of the site that I chose and why.
Let's put in the URL for the robots.txt file here.
So here we have the robot's URL.
And now we need to make a request and get the robots.txt file so we can say async function.
Let's call it get robots.txt and we pass in a robots URL.
And then inside here, we make a request where we just get the robots.txt file.
Remember, that request is just getting a file, like something similar to curl.
If you know curl already, you can run it inside the console to just download files or similar to Postman
which is making get request.
That is also just getting the file without parsing a JavaScript or doing anything advanced like a browser
would do.
But we're also going to look at an example where I get the robots.txt from puppeteer or the Chrome browser
in an automated way.
Um, so okay, I did a little sidetrack here because I just wanted to explain you guys what, what request
is all about.
So we say request.
Dot get and paste in the robot's URL.
And now we basically have the robots.txt file here because request is just getting a file.
And for the HTML we just get the HTML.
But now since we just have a text file, it's just getting the text file.
And well, then we can say, Well, then we can just.
Return robot text here and we can also just put it inside the robots parser so we can say const parser
or robots.
Const robots and we can say robot parser.
And then we can call it like this.
And in here you put in the URL so that it knows what directory you have this robots.txt file lying in,
which is what I mentioned before, how the rules are different depending on what directory it is inside.
And then we have the contents itself.
So that's why we just put in this robot text which request is getting for us.
So that's the text file directly from the site.
Okay.
Now let's try and see if we can actually make this work and test out some rules.
So let's first take a look at the robots.txt file here.
It doesn't have that many rules.
It does have a crawl delay of 10s.
So keep that in mind.
It actually wants you to wait for 10s before you make another request on this site.
And it has various user agents that it has disallowed.
And there's one here.
It also specifically says it wants a crawl delay of 10s.
Okay, so let's try and do a console log here of robots where we can say is allowed.
And we could check out if we are allowed to.
Let's let's try and find some URLs here.
If we are allowed to.
Let's see.
Read my favorite 100.
Let's see if we are allowed to get that one.
Looks like the HTML went kind of weird there, but let's see if we are allowed to scrape this side here.
As.
Let's call it the Stefan part.
Or you can call it.
Your own name.
But.
And then let's try and see if we are allowed when we are not supposed to, when we have a one of these
user agents, which they don't like.
When you have a disallow with just a slash, it basically means don't scrape anything.
We don't want you to scrape anything.
Um, and well, if they're lucky, maybe the bot respects it.
Now, let's see.
So now we test out this one in in the bottom should be false because that's Roger Bart.
And they say, Roger Bart here should not be allowed to scrape anything.
And also Stefan Bart should be allowed because it's just the directory 100 and they fortunately don't
have any rules against Stefan Bart in here.
So let's test that out.
So we say make sure to save it and make sure to call this function as well.
So we call it here and then paste in the URL that we have up here.
Robot's URL.
Okay, so that's it.
Now let's try and run the code and see if it works.
Okay, so everything looks good here.
We got a true first where it.
I also guessed it would be true because this user agent is allowed and then a false because this user
agent is not allowed to scrape at all.
So that's um.
That's really how you get a robots text file out from request and then how you paste it in, pass it
in to the robot parser and check if you the rules that is allowed.
We can also try and check if how the crawl delay should be get crawl delay.
And it should be ten because in the rules here for any user agent, it says scroll delay ten.
So remember that if you're trying to scrape this site, they want you to have a delay of 10s for every
request.
Let's try and see it again.
So it says 10s.
Okay, so now in the next section, I think we are going to actually try and crawl the site and we'll
basically have a web crawler which is going to just check for all the links on this site and go to all
of the files and basically get them.
I think let's try and see if we can do that next section.