Okay, so on to the juicy part, which is actually scraping the data we want from this these description
pages.
Let's see how we can do that.
So we actually need to do pretty much the same thing we do up in the index page.
We're just going to get some different selectors instead.
So I'm just going to be lazy and copy this.
From here down to here.
And then we have this nice jQuery selector we can use for the page.
Let's try and run it and just to open the browser so we can look in the HTML and find our selector to
get the data.
So we got a description page up here.
Oh, I closed the window.
I was not supposed to do that.
So we got a page here.
I'm just going to open it in a new tab so that my, uh.
It's not it's not going to load a new page.
Instead, while I'm looking for a selector, I'm going to open the good old developer tools here and
select an element.
And we're just going to go straight for the money guys and select the currency and the value we got
here.
And so I'm just going to right click this one, say copy, copy selector, then we're going to paste
it inside of the console and we can see the selector here.
So there is no real defining classes or IDs I can use to select Exactly.
This element is just random classes you see here, random names.
But I think the most efficient way to do this is to get rid of all the random class names and we can
still use this long list of parent child selectors and go through the Dom tree to actually get our value.
I think that's a more, um, simple way to do it.
Rather than taking all of the HTML and using something like regular expressions to look for, uh, something
that looks like this text here.
I think the, the more simple way to do it is to just go and look for this tree and find exactly the
child element you want to get.
So let me show you how we do that.
We delete all of these weird names you see here with the dot before it.
So this one, delete that.
And you also want to delete this one because they're randomly generated for each of the descriptions.
So we don't want to select by class names.
We just want to select by elements instead.
So make sure to delete all of these class names with the dot in front.
This one.
And this one.
And you should be ending up with something that looks similar to this.
Selector here.
Just a lot of this one with the child selector here, some and some spans in the end.
And we copy that.
And let's just see if our jQuery I mean, our CSS selector is working.
So we just do this and we can see it's working fine.
I'm getting the price in here, so I'm just going to well, copy this.
From the console and paste it into our script description page URL function we have here and let's call
it price per night.
And then let's do a good old fashioned console log and let's see how that's turning out.
So now we should be getting prices for each of these rooms, right?
Price per night.
Let's see.
Well, now it's showing dollars over here.
I don't know why.
That's weird.
That's really weird.
Anyway, you can see we got the first price of this one.
We got this one also.
The other one was blank.
You notice that here we get the price.
And here we got the price.
There was one that was blank.
And here we got the price.
And here's a plank.
Another plank.
But it was 350.
You saw that shortly.
Here's another plank.
Right.
So why are we getting planks here?
Well, the reason we're getting planks is because the page has not been fully loaded yet.
Now.
Excuse me.
Sorry for that.
And now Puppeteer has something.
I'm just going to go away from this.
Puppeteer has something that is called Wait for Selector.
So you could use something like Await Puppeteer.
Uh, sorry, await page.
Wait for selector and you pass in the, uh, the CSS selector like this inside.
But unfortunately, we can't do that because the CSS, uh, the element is actually being created,
but it's not being filled out until later by the asynchronous JavaScript.
So what can we do instead?
Well, we can do a trick here where we say an option we pass into the page dot, go to.
We pass in an option saying wait until that the network has less than two connections for more than,
uh, for at least half a second.
Let me just copy that straight from the puppeteer documentation, how it's it looks like.
So the wait until Network Idle two is going to consider navigation to be finished when there are no
more than two network connections for at least half a second.
You can also use a network Idle Zero.
So when there is no more than zero network connections for at least 500 milliseconds.
But this one is a little faster, but it's going to work fine for what we are getting.
So that's going to make it only consider the page to be loaded once.
We're not fetching lots of data in using JavaScript.
Inside of chromium, of course.
Now, let's see if we get lots of blank prices now or if we are actually getting some good values.
So.
720 And.
900.
So it looks more promising now, Right.
So it's going to be a little slower, but it's, of course, a lot better data you get out of the the
result.
Okay, everyone, now in the next lecture, we are going to get the rest of the values we have inside
of the rooms.