WEBVTT

00:00.230 --> 00:00.650
Okay.

00:00.650 --> 00:07.610
So the first thing we are going to do is to make a new directory for our robots parser, but we're just

00:07.610 --> 00:13.940
going to call it web crawler for now because we're going to have various types of crawlers in here.

00:14.420 --> 00:21.100
So we call it web crawlers and let's initialize NPM inside the web crawlers directory.

00:21.110 --> 00:23.540
So I'm going to go into Web crawlers.

00:24.330 --> 00:26.340
And say npm init.

00:26.340 --> 00:27.210
Dash dash.

00:30.250 --> 00:31.270
And that's it.

00:31.270 --> 00:36.070
And now we're going to open the directory inside of Visual Studio code.

00:37.510 --> 00:40.630
So let me go and go into red colors.

00:42.040 --> 00:46.750
And now we are going to import some packages and write out some code.

00:46.840 --> 00:54.790
So we're not going to build out the robots parser from scratch because that's a lot of work and thankfully

00:54.790 --> 00:59.080
someone else has built a robots.txt parser for us.

00:59.170 --> 01:07.210
We're going to use the robots parser from Sam Clark and it's got all the things we need and we're just

01:07.210 --> 01:10.960
going to add that using either NPM or yarn add.

01:11.490 --> 01:19.380
So now let's go into Visual Studio code and make sure that you open a new terminal via terminal new

01:19.380 --> 01:23.340
Terminal and add the robots parser package.

01:23.340 --> 01:26.070
So we say yarn add robots parser.

01:26.250 --> 01:29.910
You can also use NPM install if you prefer to use NPM.

01:30.690 --> 01:33.330
And then let's go and make a new file as well.

01:33.600 --> 01:42.720
Let's call it robot parser dot js and in here we're going to have some robot robots text parsing code

01:42.750 --> 01:47.850
that we can use globally in all of our projects for these coders we're going to build.