All right, So now we got the we got the CSS selector for the the poster here.
Now we want to go through each of the movies to do this automatically.
So to begin with, we are going to write a function which we call scrape poster URL, and this takes
in the array of movies we made in this function here.
So this is going to be a little weird now, just like in our but it's actually just like in our Craigslist
scraper because we're going to use a for loop and go through each of these movies in parallel asynchronously.
So what I mean by that is that we have this movies with post URL.
And we are going to use the promise dot all and we are also going to use.
So we run a movies dot map and here we have async movie which we go through each of the movies and this
is pretty much just like in our Craigslist scraper.
So the reason why we are using Promise dot all is because we're going to be using some async calls to
we're using requests in this dot map function.
And if you don't have the await promise dot all the the um it's not going to wait for all the promises
to finish and this array is not going to be complete with the data.
Um, so this is just like our previous quick scraper.
But let me show you again what we do.
So we have, um, the HTML from, uh, from request.
So request dot get and we say movie dot description URL.
So the description again is just this page.
We go inside to here and we make the.
Uh, stereo selector here.
So stereo load HTML.
And now we can select the poster URL.
Actually, I'm going to just add it to the movie here.
Movie dot poster URL, and I'm going to select the element here.
So that's the one we got previously and get the h href attribute like so.
And then I will return the movie.
So then we're going to end up with the same array as we have here in this function, but it's going
to be also adding the poster URL to each element.
And then after that, I think I also want to return the array here.
Now in this function we made over here, scraping the title ranks and so on.
Instead of just doing a console log, I also want to return the array here.
So return movies.
Yeah.
And I think also because we're doing so many requests, um.
Request.
Get in here.
I think it's a good idea to have a try.
Catch clause here and.
Just put it in here.
Just in case we get some errors.
I don't want the whole thing to shut down.
If there's one URL that is odd or something.
So doh, doh, doh, doh.
So now we are going to make another async function, which is going to be sort of called our main.
And in here.
We call the previous functions.
We wrote this one.
So this one is now returning the array return movies with all the titles, ratings and so on.
And then we put in the movies again inside of our scrape post URL, and this one is going to return
the movies array, this time with the post URL to each element.
And then we can try and do a console log, see how our array is looking.
And now instead of calling this one scrape titles and ranks, we are going to call Main instead.
Okay.
Now let's try and run it.
So I am getting a heat error here.
Movies dot map is not a function that is art.
Well, that's because I need to also await this function here.
And I also need to await this one because they're also asynchronous functions.
Let's try again, ladies and gentlemen.
Moment of truth.
So we get some errors here.
So like I said, there was some, um.
There is some errors.
We get some functions that are missing the URL or something.
We get those up here.
I can also just comment that out for now.
If I don't want my console log to be messy.
We could also look and take off any elements that doesn't have the title to make it a bit more clean.
The data we're working with, but for now I'm just going to keep it like it is.
But as you can see, after the errors we get here, we see the actual elements we have here.
We have a poster URL for each of these elements.
Also, just like in our other example, I'm going to prepend this imdb.com to the URL.
So it's just ready to go.
Let's try and.
Run it again.
See how it looks.
All right.
Now we get this nice URLs here for each of the movies containing the poster URL.
So if you go inside to this URL here.
You get to the poster itself.
See.
So in the next section, we are going to be looking at scraping with the nightmare, scraping the picture
itself, and why actually we need to be using a nightmare in this example.
Okay.
See you in the next section.