Today, we're going to be looking at a website where you have to log in in order to be able to web scrape
the content that we want to get.
And the website is also expecting you to have some kind of cookie set and also pass some data from these
cookies to the login form.
So it's a little trickier example of how to do login with web scraping.
But I'm going to show you step by step how to do this.
So first we have the website.
It's a internship website for Indian students.
I think I had one of my students ask me about how to scrape this website.
So we go on to, Well, first you need to register.
If you don't have a user inside, just use a temporary email like temp email or something like that.
If you don't want to use your own email, you can use like a disposable email to register inside there.
So go ahead, make a user inside there.
Then we go to the login form.
Let me show you what we're doing here.
I open up my chrome developer tools.
And we go to the login form and let's watch on the network tab here.
So there we go.
Now log in.
And then it shows the internship postings.
These are the postings we have the objective of trying to scrape in our web scraper.
So how do we log in using NodeJS requests on this website?
Well, I'm going to show you that now step by step.