So today I had a question from a student in the Web scraping course, John, who is asking me.
He's trying to scrape a website, but when he tries to get the URL of an image using jQuery or end request,
he gets back this base64 URL, which doesn't really make any sense.
It's not the full image that's encoded in base64, it just seems like it's some random base64 he gets
back compared to when he looks inside of the actual site using Chrome.
So if we go inside Chrome developer tools and let's say that we want to extract the images from this
new site, the abc.net.edu, and we try to look on the images here that we want to get.
Now we see here we have the image which is inside this featured card.
Um, and there is a source here with the image URL.
Let's try and click on that.
Then get selected.
Okay.
I selected it now and what that shows the image.
So that's what John from the course the student is trying to achieve, he's trying to get all of the
images we have on this new site now.
But the problem here now is that John is writing when he tries to do it inside of request, he gets
back these base 64 URLs instead, which he can't use for anything.
You can't even decode this base.
64.
It doesn't make sense, basically.
So what is happening here?
Well, sometimes a website is acting a little differently.
When you don't have JavaScript enabled or if you're if the user agent is different from the one you
have inside Chrome.
So there's two things you can do.
Well, there's actually a more than two things you can do.
Um, first thing you can try to do is to change or set the user agent header inside of request.
So you add this option object, and then you add the headers property, and then you set the user agent.
Now, what should you set as the user agent string?
Well, if you Google something like what is my user agent?
Then it just comes up right here from Google and you can just copy that string and then paste it inside
here.
And then you should have the same user agent.
Now, in this case, it didn't make a difference.
Sometimes it works, but in this case it didn't.
So what you can do then is to inspect what data or what are you actually getting back from the this
new site or this site you're trying to scrape.
So let's go ahead and try to do that now.
So to do that, I'm going to save this HTML we get with request the result variable here and then save
it into a file using the FS module.
So let's go in here and say const FS require FS, and that's just a file writing module that we can
all use inside of NodeJS.
So here we can say something like FS write file and I just use write file sync because it's simpler
to write.
And then I can say something like dot index HTML.
And then we pass in the data, which is going to be the result from the request we made.
So that's just HTML that we fetch using request.
And then afterwards we pass it into chario so that we can select elements.
But anyway, now let's try and run the file again and see what kind of HTML we are actually getting
back and see how is it different compared to the ones we see inside of our Chrome browser when we look
on the live internet version?
So let's try and run the file or let's run the node index.js file.
And then out here, I can see now the index.js or the index.html file has been created.
Let's try and run it again just so you can see how it's been created.
I run the file, the index.js file again, and now we can see the HTML file is being created from the
FS module.
So let's try and inspect this HTML a little bit.
You can either inspect it inside vs code.
Um, but since it doesn't really, uh, tidy up the HTML, it's like raw HTML we are seeing here.
It's not being formatted very good.
So we can try also to reveal it in our file Explorer and let's try and open it up with our browser.
In this case, I'm going to use Chrome and this is how the page looks like when I load it up inside
of Chrome.
Now, this is where the images are supposed to be.
And I guess something is missing here.
It could be JavaScript, it could be something else.
Or maybe it's just not fetching the images for some reason.
Um, yeah.
So the it seems like some things might be blocked because of course, or maybe some JavaScript we're
supposed to fetch the, uh, the images, whatever.
But we can try and still inspect where this image was by using the select uh, element to inspect it.
This cursor we have up here and we can try and click where the image was supposed to be.
So I click here on this div here and try to go inside another step.
And I can see already here there is an image.
So here's the image and let's try and see how it compares to the original site so I can try and see
When I click here, there is again this same image, but it it does have a difference now.
So if I try to see them side by side here, a crowd of people wave their hands while attending an open
air church rally, the same one as we have here.
We can see that the source here has a URL, a good usable URL for the image, but inside the HTML fetch
with request it says this data image base64 encoding I talked about before, which we can't use.
But if we look closer here, we can see that theta dash source property actually has the proper URL.
Now, I don't know why this happens.
Um, we can also see some no script going on here.
So they tried to make it for if you don't have JavaScript enabled.
Um.
But something went wrong here.
Maybe because it's a local file.
We can also try disabling JavaScript on this page.
So let's try and disable JavaScript on the.
On the page and see how the the the HTML looks like if you do that.
So I have this quick JavaScript switcher if I click on that.
And disable it.
Let's try and see.
So now let's disable I'm going to pin it here so I can see that it's disabled.
It's red now.
Now it's green.
JavaScript is enabled.
Now it's red.
It's disabled now.
And you can see, well, the page still looks the same, but JavaScript has been disabled and this is
the same way as request will get the page because JavaScript is not enabled inside of NodeJS request.
So if we take a look at this page or this image now we can still see the source is still the same and
so it still looks the same as our regular JavaScript enabled version of Chrome.
But for some reason or another, the NodeJS actual fetched HTML still looks different.
Why?
I don't know.
But we can see here that the data source property actually has the URL of the image so we can still
use this fetch HTML instead of using puppeteer.
If all else fails, if you can't get this image.
For some reason through Node.js request, you can fall back to puppeteer, but I would strongly recommend
against using Puppeteer as much as possible because it does take more resource and is more prone to
crashing and so on and so on.
Of course it still works fine if you have to use it at some point, but try to use NodeJS request as
much as possible.
Now all that blabbering, let's try and see if we can actually make this work and fetch some images.
So let's try and go for the data source property instead in here.
So inside of our NodeJS index, instead of going for source property which worked inside Chrome browser,
we're going to use the data dash source attribute instead.
Let's run Node Index.js and here we can see all of the different actual URLs of the images that we want.
Here we go.
So we can see.
Now we can get the images just like we wanted to.