So what if you couldn't get your request working?
When you're looking inside of Chrome developer tools, you couldn't figure out how to get the login
request or you couldn't figure out how to replicate it inside of Postman.
Well.
There is a fix for that and that is to use puppeteer.
Now, it is going to be the last option I would use because Puppeteer launches a browser, a Chrome
browser and automates it, which uses a lot more resources than requests and is a lot more prone to
crashing.
It is a bit more heavy on resources, so if you need to run it somewhere, it's going to take more resources.
So it's going to be the last option I would use.
But it is a viable option to use.
And I'm going to show you how to do it if you can't make it work otherwise.
So first, let's go ahead inside of Visual Studio code.
Make a new directory.
Let's call it Craigslist.
Puppeteer.
And then initialize NPM.
An em dash.
Dash?
Yes.
Then let's go ahead and open the project inside of Visual Studio Code.
Then let's go inside the terminal again.
And this time, let's say yarn add puppeteer.
And it might take some time for you.
If it's if it's the first time you're downloading puppeteer It's a quite a big package because it's
getting all of the Chrome browser, basically.
Now, once that is done, let's go ahead and make an index.js file.
And let's import puppeteer.
And then let's make a main function, as we always do.
And call it down here.
Let's also put a try catch clause around it just for good measure.
Okay.
Now, first thing we need to do is to make the browser instantiate the browser inside of puppeteer.
So we go const browser.
I wait for it to launch.
And then let's put in the headers and set that to false so we can see the browser.
If you're not, if you're deploying the browser on some server somewhere, then you probably want to
set this to false or just keep it by default by removing the parameter.
But while we are debugging, it's really nice to be able to see the browser.
So let's go ahead and type in Node Index.js and see what happens.
So here you can see it launched a new Chrome browser window.
It says Chrome is being controlled by automated test server test software.
So that is how far we got so far.
Now let's make a new page.
So we say const page await and then browser dot new page.
And then we simply say go to a specific URL.
So wait, page go to and then we paste in the URL of this Craigslist login site.
So I'm going to log out.
And then we have this URL here with the form to log in.
Then let's paste in the URL here and let's see what we got.
Now, so let's start it up again.
And now you can see it opens up the browser and goes to a new tab and it starts to go to the site where
we can log in.
Now what we need to do now is to type in our email and our password, and then we need to press the
login button.
So how do you do that inside of Puppeteer?
Well, it is quite easy.
We simply say await page, dot type, and then we need to put in the CSS selector for this element that
we want to type into.
So if you open up chrome developer tools.
And you use the select element over here.
Point on the form input element you want to input on.
For example, this one you can see it's a type of input and it has an element ID of input email handle.
And you could also see when I hover my mouse over the CSS selector is right up there in the upper right,
upper left corner saying input with the hashtag input email handle.
So let's use that as our CSS selector.
So then we say input, hashtag input, email handle.
Let's check if yeah, that was correctly spelled and then put a comma and then you type in the text
that you want to type in.
Now let's see how that works.
So there we go.
So now it went to the side and it also typed in my email in the login form.
You can also pass another optional parameter or option with a delay.
So if you want to have a delay on, say, 20 milliseconds or 200 milliseconds for some reason, then
you can do that.
But for now it works without.
So let's keep it simple.
Then let's say we also need the password.
So we need to find out what is the CSS selector for this password.
So again, we hover the selector over here and I can see that it's input, hashtag input password.
So we say, wait.
Type input.
Input password.
And then I paste in my password.
Which I CAN'trillionEMEMBER in my head.
So.
Oops.
Just like that.
And yeah, so let's make another test and see if it types in the password.
And it does.
So now we just need to click on the login button.
And again, it's very similar to when you use the type, you just say click instead of type.
So page, dot, click.
And then put it in the CSS selector of the element that you want to click.
Then you're going to Chrome Developer Tools.
Find out the CSS selector.
I can see that.
Let's move this away.
Here we can see this is button with an ID of login and the class is account form button.
But let's try and just say button hashtag login with the login is the ID.
Which you can also see if you.
Look in here.
Here is the element.
There's the ID and there's the class.
Let's just use the element and ID That should be enough to select this element.
So the element here and the ID.
And again, you can also set a delay here if you want to do that, but we are not going to use it because
it's not necessary in this situation.
For some sites, maybe you want to make your your scraping look a little more human like and you want
to put in some random delays.
That could happen.
But for this site, it works fine without.
So there we go.
And now it is logged into my account and it went into my account page where I can see my my posting
in here.
And we also need to go and we need to go to the billing page, which is our objective in this scraping
project.
And so then we need to log in and then go to the billing page.
So let's see how we can do that.
So obviously the most natural thing you might want to do is to say, await page, dot, go to, and
then let's put in the URL we have up here.
For the billing page.
So we go ahead and do that and let's see what happens.
So as you can see, nothing happens.
We're not being redirected to the bidding page.
We're not even logged in.
And if you were really fast, you can see that it typed in my login information.
It clicked on the login, but then it went straight to this bidding page instead of waiting for the
login to actually finish.
That's what a human would do.
We would wait until the login has finished and then we would click on the billing page to go to the
billing page.
So how do we do that?
Or how do we make Puppeteer know to do that as well?
The way is to say await page dot, wait for navigation.
And that's it.
Now, it's going to wait until this redirect has happened and the page is loaded.
So let's see what happens now.
And there we go.
So it was quite fast, but it did exactly what we wanted to.
It went and locked in and then it went on to the billing page was once the navigation has finished.
And now if we want to get this value here, then we can use Cheerio instead.
So let me just stop this one here and then let's say yarn add.
Cheerio.
Then let's import Cherry up here.
And now we can say const content.
Await page content.
And then we can say const jQuery sign or dollar sign await.
Cheerio.
Load content.
And let's find out.
Then let's find out what the CSS selector is for this building.
So if you go and click and select this element here, let's say that's what we want to scrape.
We can try and say right click here.
Say copy.
Copy selector.
Let's see how that selector looks like.
It looks okay.
Reasonable if I try it out in the console here.
Dot text.
I can see that we indeed get the text that I like to have.
So let's just copy that selector and use it inside of our Cheerio.
And then we can say that text as well.
And let's just say console log so we can see if we are successful.
Now let's try and run it again and see if it works.
Now let's check our console log and we can see no paid posting accounts exist, which means that it
indeed did get this information here.
So that is it, people.
That is how you can add as a LAStrillionESORT as sort of using a big hammer to put in the nail in the
wood, so to say.
That is the last option you can use if you can't figure out how to use Postman or request NodeJS to
log in.
Now, as I said before, it does use more resources than requests in NodeJS.
It is a bit more prone to crashing.
If you're loading up a full page, your browser can crash.
But it is a perfectly viable solution.
If you can't figure out how to do the login request.
Some sites are just really weird or you can't figure out how you can get this request working inside
of Postman.
And then at some point you can simply say, okay, I'm just going to do this inside of Puppeteer instead.
That is a perfectly fine solution and I hope you can use this to login in and scrape any other sites
you may have in the future.
So that is it, people.
If you have any other suggestions or questions, let me know.