Notice how fast the tests are running when we do this rather than when we are hitting the site itself
and writing out our scraper.
And at the same time.
Craigslist had no idea that we are building out a scraper for their site, and that's going to really
lessen the likelihood of us getting banned.
Actually, we can't get banned because we are not hitting Craigslist at all at the moment.
So then once you have written out your scraper, you can start running it at certain intervals instead.
Maybe only once an hour or once a day.
And your likelihood of getting banned then is quite small.
As long as the scraper is not scraping a huge amount of data at one point.
And you also have a security belt that your scraper is working as it should be, and you can keep on
making changes in the code and note that it's still working.
For example, if we wanted to make a separate function for getting the different details, I can do
that.
So we can write out a function, for example, for get date posted where we pass in the jQuery variable
and element.
And then we can simply say we return this.
So I can say return.
This one here.
And then down here, we can call get date posted.
And I can pass in the two variables here and I can still see that all of my test is passing.
So I have a security knowing that my code is still working, even though I did this refactor right now.
So that's going to enable you to make changes in your code and ensure that it's easier to maintain.
Let's try and get one for getting the hood also.
So we write the same thing basically.
And then we cut this for getting the hood.
We say return this one here and then we call get Hood and pass in the dollar sign and element.
And I still see all of my tests are passing, so I know that the code is still working.
And I will have to tell you, that makes your code in the long in the long end a lot nicer and easier
to maintain rather than having some code that you build out and suddenly you don't want to touch it
because you don't know if you're going to break anything.
So there's a lot of, um, there's a lot of positive.
Effects or side effects and lots of positive things about doing TDD.
The only negative, if you can call it a negative, is that it does take extra work.
We did have to write out all of these these tests.
We had to.
We had to figure out a way to read this data without hitting the actual site.
So you have to do some extra work when you write tests.
But hopefully and it usually does pay off in the end.
So that's been a little introduction to you guys for test driven development and how to build a scraper
without actually hitting the site all of the time.
If you thought this was interesting, let me know if you'd like to know some more things about how to
do this.
I'll be happy to make more examples.
So see you guys and I hope you enjoy the rest of the course.