Now here's a question.
Have you ever gotten into a situation where it's evening time and you wanna
watch a movie, but you don't know what to watch?
Like there's just no ideas coming to your mind and everybody's sitting around
the TV, like, what do we do?
So what if we scraped a list of the top 100 movies of all time and you pick one
from there?
So I recently came across this article on empire where they list the 100
greatest movies of all time.
And you can see that it goes from 100 all the way down to one.
These are supposed to be the top rated movies that have ever been produced
and I always wanted to watch through all 100 of them and check off each one as I
go along. So this is what we're going to be doing for our final project today.
You're going to be scraping from this website the 100 movies
and you are going to be using Python code to create a text file called movies
.text that lists them in order, starting from one.
Notice how each of these are just the titles of each of these movies.
Essentially, it's this part that you want.
And because the listing starts from 100,
you have to figure out how to flip it the other way so that you get it
starting from one. But essentially this is the goal.
And this is the project for today.
You're going to have to use what you learned about web scraping,
but also other things that you've learned along the way about Python.
Pause the video now and try to complete this challenge.
1: Okay.
0: So here I've created a brand new blank project,
which I've called top100-movies, but of course, that doesn't matter.
But in the main.py is where we're going to be doing all the serious stuff.
So the first thing we'll need is to copy the URL.
And at the time when you're doing this project, this URL might change.
So don't get it from here. Instead,
go to the course resources and copy the URL from there.
Once you've got the URL,
we're going to paste it into our main.py and save it as a constant.
In addition, we're going to have to import all the things that we need,
including requests,
and also the bs4 module
where we can get hold of our Beautiful Soup.
I'm going to need to install both of these because I don't have them yet.
So I'm going to click on the red squiggly line, install requests,
and also install beautiful soup. Once that's all done,
then my red squiggly lines should be gone and I can now start using my request
module to
make a get request to this particular URL. And I'll save the response
I get back as the response.
The actual HTML files,
so the website HTML, is actually under response.text.
So this is going to be the raw HTML text,
and this is what I'm going to be using to use Beautiful Soup to parse.
So now we make soup and we're going to use Beautiful Soup and we're going to
parse in our website HTML
and also the html.parser
which is the method which we're going to use to parse through this website.
Now that we've made soup,
let's go ahead and print out our soup in a predefined format so that we can just
see what it looks like.
So let's run our main.py and we can see that we've got basically that
entire website's HTML being printed out
here. Now comes the part where we need to use our Google Chrome inspector.
The part that we want from this website is just these lines. Now,
of course, if we didn't know how to code,
we would be here copying and pasting for hours on end,
and we would die of boredom or we'd get repetitive strain injury from doing so
much copy and pasting. But because we know code, we know better than that.
So let's get hold of the part that we want. Let's right-
click and click inspect. Now,
we can see that this lives inside and h3 with the class of title.
Now let's just check against one of the other pieces that we want and just make
sure that they've got the same structure.
So this is also inside an h3 with the class of title.
So basically as long as we can scrape this entire page and get all of the
h3 with the class of title and get the text that's contained inside the
h3 element, then we're golden. Let's go ahead and do that.
So instead of printing our soup.prettify,
I'm going to tap into soup and I'm going to say
find all. The thing that I want to find has the tag name of an h3 and it
has the class of title.
So that all came from our inspections right here.
This should get us a list of all of the
h3 elements with this class
and we should be able to save that into a variable called all_movies.
Let's print all_movies and see what we get.
Now, we've got a list of all of our h3s,
and we're now going to go one step further and fetch the text from within these
h3 elements. We do that using the
getText method. But we can't do that on the list
so we're going to use list comprehension.
So we're going to say the movie_titles is equal to a new list
and in this list,
each item is going to be formed from a movie in the all_movies list.
And this new item is going to be created by taking each of the movies in the
list and then calling getText on it. Now,
if I go ahead and print my movie_titles
instead of all_movies, then this is what we get.
We get all of the titles of all 100 movies.
Now we want to reverse this list so that we can put it into a text file starting
from 1, going down to 100.
So there's a couple of ways that we could do this.
One is we can use the Python splice operator.
So we add a set of square brackets
and then we add a ::-1,
and this will reverse the order. And as always,
you can find out this information either by Googling or through what you've
learned before in previous lessons. So this is the syntax that we're using,
which comes from the slice operator where we have a start,
a stop, and a step. So in this case,
the start is at the very beginning of the list,
the stop is at the very end of the list so
we don't have to specify those cause they're the defaults,
and then the step is basically -1 and this syntax will reverse that list.
Now, alternatively, you can,
of course also use a for loop where you create some sort of n in a range and the
range again, can take a start, stop and step.
So you could start at the end,
so the length of our movie_titles-1
because remember lists in Python start numbering from zero.
So the very last item is actually at the total number minus one.
And next is going to be the end, which is going to be zero, and finally
it's going to be the step, which is minus one each time.
So this time we start off from the very end of the list,
go back to the beginning, stepping by minus one each time. And this way,
if we print n,
you can see that this is going to give us basically all the way from 99 down to
1.
And we can use that to tap into our movie titles and get hold of each of the items
at index n.
And because range actually doesn't go beyond the end
we actually have to put -1 there if we want to get the very final one
which is number 100. And you'll notice also that there's a
bit of a typo here for number 93,
and this is actually not our fault at all.
It's in fact, in the original empire post.
They actually screwed up and this should be number 93. Because we're scraping
data we can't really be picky.
We're just going to end up with what we end up with.
So I'm going to choose the method where we actually take the movie title
and then we use the splice to get hold of it in reverse.
And I'll call that the movies.
And now we can create our new text file. So with open
we're going to create a new file called movies.txt,
and we of course have to change the mode two write mode so that we can actually
create this file. And then
because this file movies.txt doesn't exist, once this line runs
it's going to create that file and then we're going to write to it.
So we're going to say file.write
and we're going to write each of the lines in our list of movies.
We can again use a for loop
so for movie in movies,
let's go ahead and write the name of the movie
and then lets add a new line operator,
so \n, so that we get each of the movies onto its own line.
Now, finally, if I run this code,
then it's going to create our movies.txt. And if I take a look at it,
you can see it's now got all of the 100 movies listed on here from 1 down to
100.
And now you can go through this list and delete the ones that you've already
watched and then continue watching through the rest of the list.
I hope you had fun trying out web scraping by yourself in this project.
So that's all for today. Take a look at what you've done.
If there's anything confusing about the class or ID or HTML,
then be sure to review some of the lessons in the previous four days.
That's all for today.