So now let's move on to Visual Studio code or whatever editor you prefer to use, and let's start making
our project with NodeJS and request.
First, let's initialize a project.
So let's go ahead and make a directory first.
Let's call it Craigslist of scraper.
Let's go inside the directory and initialize NPM so we can download some packages with npm init dash
dash js.
Then let's open that directory inside of visual studio code.
And let's go and say Craigslist or scraper.
Okay, so now we have the off script opened inside Visual Studio code.
Let's go ahead and make an index.js file.
And then let's let's also open our terminal so we can import some packages because we're going to need.
So we say yarn add.
We're going to need request.
And request.
Promise.
That's it.
So now let's import request up here.
So require request promise.
All of this is basically something you already know from the previous Craigslist tutorials.
And now let's say we have a async function main.
And then let's run Main down here.
Now let's try and make our request.
So const HTML await request and then we say post because we are making a post request.
And then we paste in the URL that we found from the Chrome developer tab and used inside of Postman
and paste it in here.
And then let's also put this inside of a try catch clause.
So any errors get logged?
Catch error and say console error.
Error.
Now we need some way to pass in this form data.
If we look inside of the request promise documentation, if you scroll down a bit, look around, you
can see there are some sections about how to post data, and there's also one about how to post like
HTML forms too.
And the way they do that is you simply set this form object with keys and values and they're going to
set the header of content type to be this zvb form URL encoded.
So that's basically what we need to do.
So let's put a little comma here.
Curly braces and then write form.
And then another curly brace.
And here we pass in the key and values of this form data.
So let's go ahead.
Look inside of Postman inside the Body.
And then let's paste the Or copy the key here.
Input email handle.
And paste in the value.
And the same for the password input.
Password.
And let's copy the password.
So now we have all of the form data.
Next up is the headers.
Remember, we had to set a referrer header in order to make our login request successful.
So go ahead and copy the value of the referrer header and then make another comma here and make a headers
object and put in the referrer and the value.
So now we set a referrer header.
Inside of our request.
So that's all we needed to make it work inside of Postman.
Now let's see if it works inside of request.
So to see if it works, let's try and say import F.
S just to save the result and check it out once it's done.
So write file sync and then we can check if the login actually is working.
So let's try and run Node Index.js.
And now it doesn't look like it actually saved the file, which means it probably never came to this
point.
And it also threw out a huge response object, which means it probably threw an exception instead and
did a console error.
And the first thing I always try and look for is the response code.
So let's look for the response code.
We have status code here of 302 says found, which is actually the same as we have inside of the Chrome
developer tag.
I mean the chrome developer tools.
Interestingly enough.
It is also you to found so that it's not a bad sign that we get it though we do get a 200.
Okay.
Inside of postman.
But let's try and work around this somehow by default request is going to treat any other response status
code than to x x.
2XX as an error.
So if you get something like 3 or 2 like we did just now, it's going to treat it as an error and throw
an exception.
So therefore, we never get to this point where we can check out the actual response HTML.
So in order to disable this check for if you have a.
Well, if it's going to throw an exception.
We need to make another option that is called simple and set it to false.
You can read up about this also in their documentation.
If you try and search a little for.
And let's try and search well.
It also says up here almost at the top by default, Http response codes other than to x will cause the
promise to be rejected.
This can be overridden by setting the options symbol to false.
So that's simply what we are doing.
We are setting the simple to be false.
So it's going to not treat this 302 as an error and throw an exception.
Now let's try and see how the response looks now.
So now you can see it did go further because it did not throw an exception and it started writing the
log in HTML.
So as you can see, it is disappointingly empty.
This login.html.
Now, another thing that could be causing this empty login HTML is that we are not following the redirects
that the server is trying to make us do.
By default, postman is always going to follow redirects and of course, the same inside of Chrome.
But by default we request does not follow all redirects.
Let's try and search for redirect in here.
And there's no no results for redirect inside of request promise documentation.
But if we look inside, just request documentation, which is what request promise is based on.
And try to search for redirect.
It says it's follow follows redirects by default.
But there is some kind of.
Catch this.
It is only Http 300 responses that it false redirects for.
So well.
Even though we get a 302 response it I'm just going to try and set follow all redirects to true anyway,
see if it works.
It could be that it makes a difference.
Let's try and see again how index HTML looks like.
So now we actually get some kind of HTML by setting the symbol to false.
So we moved on to writing the file and setting follow or redirect to True.
We also got a file being written.
But it doesn't look like we are actually locked in yet.
It still says Craigslist account login.
And if I go down and look, I can still see it still looks like it is asking us to log in.
It's not the actual account page I'm seeing here.
There's also something about creating account.
This is just a login page we're seeing here.
One last option that you can do is to enable the cookie jar.
So cookie jar, by that I mean the storage of all your cookies.
Postman is saving all of the cookies that the request wants you to save.
So it's going to save those.
And the same for Chrome developer tools as well.
I mean Chrome browser.
So let's try and set saving cookie jar or saving cookies to be true.
You do that by simply saying jar True.
So let's say Node Index.js again.
And let's look at the file.
And ladies and gentlemen, it looks like we are locked in now.
So here you can see home of Stefan Hulshof.
That is me.
Logout.
Postings, drafts, searches, settings, billing and so on.
So that is actually the site if I try to log in so you can see it again.
So here you can see the site when you're logged in home of Stephen Hudson with the menu up here.
Drafts, searches, billing.
Remember, our objective is to go to this site inside of billing.
So how can you do that?
Because you can't just do a post request on this side.
Really?
Maybe you could, but sometimes you have to log in first and then navigate around in the side.
So how do we get on to this billing side once we are logged in inside of request?
Well, it is fairly simple to do it once you got this done.
The thing we need to do first is to make sure that we have this cookie jar enabled for all of our requests.
So it's going to save all of the cookies for any other requests we make after this login.
So go and remove this jar up here.
The jar through.
And then up here where we imported, let's say, dot defaults.
And then in the options we say char true.
So now this request object is going to save the cookie jar for any request.
I mean it.
It can use the cookies for any request you make.
So after we log in up here, we can do another request down here where we say const billing HTML maybe
and say await request dot get this time because we're just getting things and paste in the URL of this
billing side we have here.
And then let's try and do another file, right, to check how our response looks.
See feeling dot HTML.
Peeling.
Just like that.
And now let's try and run Node Index.js again.
So now our billing dot HTML file got written.
We are still logged in.
I can see.
And our billing.
How does that look like?
It says, how does it look?
Says here paid posting accounts.
No paid posting accounts exist.
Which is exactly what there is inside of this billing page.
So now we are successfully logged in.
We can go into any site inside once we are logged in and that is because that the request object is
saving this cookie that we get once we log in.
This session cookie that I talked about earlier, which tells Craigslist that we are logged in and we
are authenticated.
So that is it, folks.
That is how you make a login with request.
How you find out where you need to log in, what data you need to pass on to log in, and how you make
the request inside of NodeJS request.
So the process to do all of this is to first.
Look inside your network tab.
Try to make a login.
Find out what log.
Here is the login itself.
When you want.
Then once you find it, try to make the same request inside of Postman until you find out what headers,
body, whatever is necessary to make this request work.
The login work.
And then once you're done with that, the next step is to make it work inside of NodeJS request.
As you could see, there is also some things you needed to change even though we had the same headers
and form data because request is a little different from Postman, it's a bit more raw or simplistic
you could say.
So that is it, folks.
I hope that you can use this for a lot of things.
I have used it for a lot of things and I hope you have lots of good, productive work out of it.
So thank you for watching.
And in the next section, I'm going to show you also how to log in.
If you can't get this to work, if you can't for some reason, figure out how to make this work inside
of Postman with the Chrome developer tools.
Or maybe the site is a more JavaScript site and you can't use an API.
The last option to do is to use puppeteer, which I'm also going to show you how to use.