Now I'm just going to copy the example that they have inside of the GitHub repository for the the package.
So just so we can get a short intro to how this parser works.
So I'm just going to paste that in here and then we can talk a bit about how this all works so well.
First, we have the import of the package up here with require, and then we have where we call the
robots parser here where we put in the URL, where we get the parser from.
And then what we paste in the contents of this robots.txt file.
And they made it into an array where they then join it with the new line.
But you can also just pass in the array from the text file and I'm going to show you ways that we can
get this text file automatically from the pages that we are crawling.
And don't worry, we are going to build this up from scratch as well.
All of this.
But I'm just going to go shortly through how this works.
And so here we have the rules from the robots.txt file.
And then down here we can then call this robot parser and check out a product programmatically.
If we are allowed to pass or I mean scrape certain pages on the domain.
So for example, this will return false because test dot HTML is inside of this allow here.
So let's try and see if we can say console log around this statement we have here.
And say node robot parser.
And then it's going to come up with an error about robots, get preferred hosts get preferred host is
not a function.
And I'm not sure why they have this function in the example when it doesn't work.
But since it's not a very used function inside of robots.txt anyway, then I'm just going to remove
it.
Problem solved anyway.
Now let's try and run it again.
And we can see it returns false here.
And okay, so one more thing.
So we had the allow and disallow on various directories and it also has a star on the agent, on the
user agent here and in here.
You can also pass in as a second argument the user agent that you are using.
In this case, they're just making it the name, same spot.
You can call it whatever you want to your crawler.
Um, but it's also to test out if there's rules for specific agent or user agent.
Now let's see.
There is another property here, which is sitemap.
Sitemap is sort of like a directory of all the pages you can have on a site.
And we have one example with Google.
They actually have a sitemap here in the bottom of their robots.txt.
So if I go ahead and go into that XML file.
They have this various directory of all the pages that they have.
If we go into maybe this one here.
And going to another page here, it's basically a list of different URLs they have that your web crawler
can visit and start crawling or scraping.
And there's also other ways to crawl a website just simply by going to a website and looking for links
on the website and then visiting those links.
We're also going to look at that method.
So that's a short, short intro to robots parser.
First is the URL to the robots.txt file, but it doesn't actually download the robots.txt file.
It just uses this URL here when it is checking for what directories we can allow.
Because if you had your.
Well, if you had your robots.txt file somewhere else, maybe you could have it inside this directory.
Then the way the rules come out is going to be a little different because you have your robots.txt in
a different directory.
So it's not because it's going to automatically download the file that you pass in the URL.
It's to make sure how to pass the rules.
You can make out a little test here.
So we write out the deer here and then let's make a new disallow rule here.
So I write disallow.
And we can say hello dot HTML.
So I just added a new disallow rule here for Hello.html.
And then down here, let's try and check out if we can get Hello.html from the base directory.
So it's not inside the directory here.
But let's try and see what it says here.
And now it says true.
So that means it is allowed to get a low dot HTML from the base directory here, even though it is saying
that it's not allowed from the robots.txt and under the directory here.
But if I put if I say let's say we say dear slash hello.html and then let's run it again.
Now it says it's false.
So now it's not allowed to crawl or scrape this site.
But if I put a slash in front of Hello.html, it means that any page that is ending on Hello.html is
not allowed.
So there's various rules here for that.
And so now it does it's not allowed on both the base tier and inside the sub directory here.
So there's a lot of rules here to keep an eye out for and that's why we have this robots.txt parser
to do it for us.
It's just something I just wanted to point out to you to keep in mind while you are building this.
Now let's go on to where we start out building our own example from scratch.
Where we are going to pass an actual robots.txt file from a real site.