So now we got all of the job titles and the URLs to the job descriptions into one nice little array
with all the objects.
But what about the job descriptions as well?
The ones that you see when you actually click on the job title or the URL for the job.
That's what we're going to get in this section.
We're going to be scraping all of the text we have in here with the job description for each of the
jobs.
So let's go inside of the code and see how we approached this.
So first of all, I think it's a good idea to call this function something else, because we're going
to have a separate function for scraping the jobs itself on the index pages and the job descriptions.
It's nice to kind of separate things in this way.
So I'm going to call this scrape jobs from index pages.
And then I'm going to make a new async function called main.
And let's call Maine down here instead of the other function.
And then we call the function scrape jobs from index pages.
And we return the all jobs from there instead.
Just like this.
Now make sure to also return all shops inside of this function.
This one.
Next up, let's create our new function for scraping the job descriptions.
So let's call it.
Here.
Let's call it scrape job descriptions.
Make sure it's also an async function.
And then what we're going to be doing in here.
We're going to receive all the jobs in here, and then we're going to iterate through all of the jobs
and get the descriptions.
We can do a dot map over here, and we have a job object for each of these jobs.
And then in here we can do a Axios request and get the HTML with the job description.
So we can say job description.
Page and we can say await.
Uh, make sure we have an async in here as well, because we're going to be using await inside this
loop.
So we say await and we can say Axios dot get.
And we can need to say the root of the URL first.
So which is prefix list dot vercel dot app slash.
And then we can just do a plus with the job URL.
And then after that we need to create this cheerio object or the cheerio selector jQuery selector.
Where you put where you say cherry dot load job description page.
Make sure it's dot data.
That's the actual HTML.
And then since let's check out how this page actually looks like.
Well you can see here there is just one div element on this page.
It doesn't have any class or ID, but we can simply just select a div element on this page.
And we're going to get the job description out from that.
So let's do that.
We can just say const description.
Use our stereo selector and say, just get the deer element on this page and get me the text of it.
Then we can do a console log of the description.
And see what we get out from this.
Make sure to call the scrape job descriptions down here.
Let's call it with the all jobs array we got from our previous function we just wrote.
Okay, let's try and run node index.js and see if we get some job descriptions in the console.
And there you go.
You can see that we get a lot of descriptions inside of the console, which is awesome.
Now in the next section we're going to also combine it into this data object we are passing around so
that we get just one object data object with both the job URLs titles and the description.