The first thing we need to do is to clone the repository I have of the code here.
I'm going to leave a link in the description or the lecture notes where you can get the scraper.
So we go ahead and say git clone and put in the URL.
And then we simply go into the directory.
We open it inside of Visual Studio code.
And it should be named cos scraper.
There we go.
And then we go and say npm install or npm I to get all the different dependencies we have.
And now the thing the first thing you need to do is to replace this URL inside of config, inside the
config folder with your own MongoDB connection URL that you made.
So go inside of lab if you use M lab and create a new database.
So go inside M-lab and in MongoDB deployments.
Create a new database.
And call it something like Craigslist cars.
I'll just show you how I made mine.
I have my Craigslist cars, Craigslist cars here.
And initially you don't have any collections.
I will delete them now and go ahead and make a user.
Add a database user.
And then you're ready to go.
So then you have a URL with a user and a password, and then you can replace that URL inside here.
So fairly basic MongoDB stuff.
If you use another provider, you can do that.
Just replace this, of course.
Now, inside of the code itself we have up here, we have the model, we have request.
Cheerio.
You should know that from previous parts of the course creates this card, the model itself that we're
going to save to the database.
It's basic model with a URL where you can see the cars article or post the timestamp.
The post was made and the title of the card article itself.
Then we have the URL where I get all of the different cars from.
That is the one I showed here in the other episode.
And so here's all the cars we scrape and well, then it just basically goes and gets the different elements.
I think I've shown that in other sections, so I will not go into too much depth about that.
And then we have.
Inserting cars.
Craigslist cars in the MongoDB where we go over all of the results.
This array of results we have from Craigslist, we just basically check if there's already this car
in the database.
So if there's already one car with this specific URL, we're not going to save it in the database so
we don't get duplicates.
And then I just well, fire off all of this requests to MongoDB and wait for the promises to resolve.
And once the promises have resolved, I disconnect from MongoDB and the program is closing.
So if you run it just by saying node index.js.
It will see connected to MongoDB.
And.
Let's give it a little while.
And then it says disconnected from MongoDB.
And now if we check inside of our lab.
Inside of the collections.
You can see there's 120 documents being made because we didn't have any cars in the collection before,
but now it's been filled up with the first page.
But if I run it again, it should not create 120 more because we already have the cars inside of the
database.
So if I go ahead and refresh the page, I can see we still have 120 documents because there hasn't been
added any new cars in this page yet.
In the one second I just waited.
So instead we wait ten minutes before running the scraper again inside of Heroku and then we gradually
add new cars that we.
So in the next section, let's see how you can deploy to Heroku and how you can run it in periodic times.
So but remember to replace this URL with your own or else it's not going to work.
And then I'll see you in the next section.