In this section, we're going to take a closer look on robots.txt.
Why sites have a robots.txt, what they are good for, and why we should try and follow them to avoid
getting banned on a site.
So we have a couple of examples of robots.txt text here from GitHub, Google, Craigslist, Facebook,
LinkedIn.
And basically the robots.txt file is saying to web crawlers what they can do and what they cannot do.
So here we have Facebook robots.txt.
Well, first of all, they have a they have a notice about they have prohibited crawling Facebook unless
you have written permission.
So there's that.
And then they have written out specific rules for the different bots.
So there's one for the able bot and then there's one for Baidu bottle Spider, and then we have one
for Bing, and then there's one for Google as well.
And then there's various rules in here, for example, disallow, which means that the Google Bot is
not allowed to scrape any directories or files or pages under the folder and or scrape anything with
album dot PHP or checkpoint and so on.
So that's one example.
Then we also have Craigslist in here.
They have a star under the user agent, which means that any user agent is not allowed to go to these
directories.
And so you can't scrape anything that ends on your chest that goes like the chest here or flag and so
on.
Let's see if we have some other interesting rules here.
Here is the Google robots.txt.
And in here we can see for all user agents, they say they disallowed the slash search, but they actually
allow search slash about and search static.
Another thing you also might see inside a robots.txt is how the interval you can have between scraping
a site.
So let me see if I can find any examples of that in here.
I don't think I could find any.
Um, but I think there is a example in the parser we're going to use inside NodeJS here.
There's one here that says crawl delay one.
So that means that you have a delay of one second for each site that you're scraping inside of a specific
domain.
So you can't just hit up a lot of pages on a certain domain in very little time or else they are not
going to not going to like that.
And maybe they will ban you.
We noticed that they had specific rules for different user agent.
They define that up here and say what user agent can do what.
But usually they have the same rules for all the different user agents.
So the user agent string is just a string that we can define inside of our crawler.
And last thing is that it is of course voluntary from your side in the crawler, if you actually want
to respect robots.txt, it's not a technical requirement when you build your web crawler that you have
to respect robots.txt.
But if you don't, then maybe this side is going to block your IP or ban you from crawling the site
at all.
So if you want to establish, I can I guess you can call it a long term relationship of crawling a certain
site, then you should probably follow the robots.txt.
So now in the next section, we're going to look at how we can programmatically parse and follow robots.txt
rules so we can test out if our crawler is allowed to scrape a certain site and then we can let our
crawler pass and get the HTML if it is allowed to pass a certain page.