So now I think it's a good idea for me to show you where do we actually run this scraper that we're
going to build because we're not going to run our scraper inside of Chrome developer tools.
We're going to run it on a server side script inside of NodeJS.
So that we can basically deploy to any server anywhere or run it at any time without having a browser
open and manually pasting or typing our commands inside of console tab inside of chrome developer tools.
So in order to get started on that, you need to open up Visual Studio code.
Hopefully you already have it installed and make sure you also have NodeJS installed.
So in here, go ahead and make a new directory.
We can call it scraping intro.
And then go inside the scraping intro directory.
And in here you need to run npm init test dash.
Yes.
So you can get npm packages and use them from node.
And then you need to do you need to add two packages.
You can either use NPM install or you can use yarn add, which I prefer to use.
Then you can say yarn, add request and request promise and then cheerio.
So let me first tell you about these two packages before we move on.
The request package and request promise is a package two packages that we use to simply download a website.
It just downloads a website similar to something like Curl.
If you know the Curl command for just downloading things or something like we get from Linux.
It's just a simple downloader for things.
It doesn't pass the site like our browser does.
It just downloads the site.
And cheerio is something we use to be able to use jQuery selectors inside of NodeJS.
So I think it's a good idea just to show you in practice how all of this works.
So go ahead and open the new folder you just created, Scraping Intro inside of Visual Studio Code.
Now that you open the folder, go ahead and create a new file and let's call it Index.js.
And in here we need to import the two modules that we imported using yarn add before.
So we write const.
Request.
Require.
Request promise.
And then say const cheerio require.
Cheerio.
And then we are going to have a main function that we then execute when we run the script so we can
call the async function main.
And then we call Maine down here below.
And now inside of Maine, we do a concert where we get the HTML of the site.
So we say await request, get.
And now make sure that you use the URL I provided earlier in the course.
So this is making a get request for this URL.
Now there's different Http methods you can use.
You can use something like post or put up update and so on.
The ones we use most of the time just to get a website is dot get.
So just like you would visit a website, you just do a request dot get to get a site and then we can
well, we can also save the HTML so we can see the HTML.
So we can say const FS to save a file, require FS, and then we can say FS, right.
File sync and say.
Test dot HTML and put in the HTML.
Then you can look at what we actually get from this site if we run this script.
Now let's run it inside of the terminal.
So I'm going to go into new terminal.
And then I will run Node Index.js.
And now you can see it has downloaded the HTML and it has it right in here.
So that is the.
That is the HTML that I have on my site over here.
So basically, it's just getting the page, downloading the HTML.
But how do we also get the data or the text that we have inside here, just like we did with our console
log here?
Well, that is where Cherry-o comes in handy.
So we can write something like.
Const Dolla sign.
Cerio.
Wait.
Cheerio.
Load and then pass in the HTML.
And now we can select all of the different elements on this side, just like we did inside of a chrome
browser.
So we can say H1.
Here with the dollar sign dot text.
And we can also save it to a variable.
So the text I have here and then we say console log the text, and then we should see the text of this
H1 element.
If I do a node index.js And there we go.
Now we see the text of this element that we scraped from the side.
And then of course, there's endless opportunities to what you want to do with this data.
Sometimes people want to save it to a database or MongoDB database or something like that.
Or well, they just want to save it to a CSV file.
There's lots of things that you can do with this data and use it for and but that is basically from
top to bottom of how we build a web scraper.
Of course, things gets more complex along the way, but I'm going to take you through different projects
where you can see how we do things.
So the reason why we're doing it inside of NodeJS in the end, even though we actually did it inside
the Chrome developer browser first, is because, well, we are going to run this on a server every
once in a while in a timed interval, maybe once an hour.
If not, then you have to go back to your computer every once in an hour and paste in the text and code
in the browser.
You're not going to be able to navigate around or go to.
It's not so feasible to write large chunks of code inside of this Chrome developer console.
So that's why we're doing it inside of Request or NodeJS.
Also, if you have to save it to a MongoDB database, you mean you can't really do that inside the console
tab in here?
So that's why we test out things.
We try to work out what works inside the Chrome developer tools.
And then once we get, for example, a selector up and working here, we which we see is working in
the console, then we go into our NodeJS code and write it out and execute it and make a actual scraper.
So that's a short intro from top to bottom of how you actually build a scraper.
You just build your first NodeJS Web scraper Now.
Congratulations.
Now I'm going to take you through on to the next lessons where we go through a different CSS selectors
that you can use to select elements.
And yeah, so I'll see you in the next section and I hope you have got something out of this so far.
I hope you build your first scraper already now.