We are now at the stage where we are fully set up and can start coding.
In this lecture, we will go over how we can replicate what we just saw in the API explorer using Python
libraries to perform these API requests.
Two final things that I wanted to mention before we start is that at first we will only use Python to
extract the data.
However, when we come to develop the whole ELT pipeline, we will need to make some changes to integrate
Docker and airflow into the code.
Secondly, we will be making requests to the YouTube API.
Each request has a cost which depends on the resource and method.
We will only use the list method, which is at the time of this recording.
For most resources, one unit and the daily limit is 10,000 units.
And here you can see an example for the resource we have used already.
Videos that lists only incurs one unit.
This is important to know.
So if you are testing and playing around with the API, you are aware that if you run too many requests,
you can reach the limit and won't be able to continue with the API requests for that day.
What you see here is the link to the quotas and the limits, and it will be in the appendix of the section
referencing what we had discussed in the last lecture.
We said that the first hurdle, which in our Russian analogy is the biggest hurdle we said it is to
get the playlist ID.
So here we need to build a script that gets it ID for us.
If we were to translate what we discussed in the last lecture into a Python script, we would first
need to use a popular module called Requests in Python that allows you to send HTTP requests very easily.
We would first start by installing the requests module dependency using pip.
You can do this by first making sure you are in your virtual environment.
Remember that to do this, we need to first run the activate command, which was this command in PowerShell.
As you can see, the ramp has been activated.
Once we activate it, we then have to run the command pip install requests.
But before we even can execute this command, I actually forgot to git push the ignore file.
So let me do that right now.
So git add git commit and let's write at dot git ignore back.
And push.
Okay now the git ignore has been successfully pushed.
So we can move on to the command that we wanted to run which is pip install requests.
Give it some time to install all the dependencies.
Once requests has successfully been installed, we can verify that the install was successful by just
doing a list command, and we will see that the requests is now showing up here.
Now let's create a Python file which will have a.py extension and let's name it video underscore UI.
As we ultimately want to get the video statistics of each video of the channel.
So let's do that video stats UI.
And the first step which we will do is to actually import the requests library.
Now we need a URL which we will use to make the requests.
Going back to the API explorer.
This time we will need to use the channels resource as we have already seen in the last slide.
To get the URL that we need, we go under channels list api fullscreen and this is the base URL that
we will use.
Obviously we have to fill in the required parameters.
In our case we will need to use content details and also the channel handle which is MrBeast.
As you can see, once we do these changes, the URL will change as well.
Now, to get the channel handle for the Mr. Beast channel, I showed you that this Mr. Beast.
But you can confirm this by going on Mr. Beast's YouTube channel and you will find the handle after
the At symbol here.
So this is the channel handle.
And as a side note here, you will also see the total number of videos at this current point in time
that Mr. Beast has on his channel.
What we can do now is to copy the URL from the API explorer and set it as the URL variable into our
scripts.
Let's do that right now.
So we're copying the URL from here and set it as a variable into our script.
Obviously here we need to replace the API key that you see here.
This we can do it by setting a variable as a global variable.
And referencing it using Python f strings.
Apart from the API key, it would be also a good thing to do that we replace the channel handle with
a variable as well.
And same thing.
Use the curly brackets.
And here we have the URL which we will use with the requests library.
Now when we use the request library we can use the get methods to get a response by writing the following
response equals requests.
So we're using the library for just imported using the get method referencing the URL.
Let's test this out by printing the response.
And this should give a Of 200 response.
Let's run this.
And we have a 200 response.
Fantastic.
You can then parse this response using JSON.
So we would write the following.
Let's create a data variable and set it equal to the response.
I'm getting the JSON from that response.
Now we can print data and showcase the output like that.
Or we can also import the JSON module.
And use JSON dot dumps.
This will take the data and we use the indent equals four.
So for those of you who are not familiar, Json.dumps method is a Python method used to convert a Python
object into a JSON formatted string.
We use indent form as it is a common convention in Python for code readability.
If we had to print this output.
Run the file.
From here we will see the JSON output that we require.
Now we said we want the playlist id which if we look at the docs you would know it is this one here
near uploads.
Don't confuse it with this ID as this ID is the channel ID.
So now it's a matter of going down the chain of JSON to extract the playlist ID.
To help you better visualize how this JSON is structured, you can install an extension on VS code.
I use JSON crack which is researching the extensions over here and type in JSON crack.
You will find the extension I'm referring to, so if you don't have it, install it.
I already have it, so it's already installed.
If at the time you're watching this lecture, this extension is no longer available or on your ID,
you don't see it.
You can try searching for another similar extension.
Once you have printed the response, Which is this one here?
You can copy and paste it into a JSON file.
Let's do that right now.
Go back here and create a JSON file.
Let's give it the same name as the Python script, but we use JSON extension.
Once you open a new JSON file, it should have this orange icon in the top right, which relates to
the JSON extension which we just installed.
Press on it and here you can see the JSON visualized.
We zoom in a bit more.
We can see this part over here relates to this part over here.
And the breakdown of the JSON structure is as you're seeing here.
A cool part about all of this is the JSON crack also gives you the path from the root of the JSON,
which you can see once you click on the uploads ID box.
So this is the full path to get to the last part of the JSON structure.
And the root in our case is the data variable, which we defined in our code, which has the whole JSON
response.
So we had to go back to the code and paste it in here.
We can change the route with the data variable.
And in order to get the value from the uploads key we also need to append the dot uploads key.
Since we're using Python and since this is effectively a Python dictionary, you can access the items
by using square brackets instead of these dots over here.
So this same structure can be written as follows.
These two are identical, but Python only recognizes the structure here and here we can remove the output
provided to us by JSON crack.
Let's structure this by splitting what we have here into two steps.
We can get the first element of channel items by defining the variable channel items.
And from there we get the channel playlist ID referencing the channel items variable we just created.
So let's create the variable channel playlist ID equals just create it here.
And that's it.
Playlist ID we should get what we are looking for.
Let's run this code.
And here we have it.
This is the channel playlist ID we've been looking for.
So at this stage we have the basic code to get the playlist ID, but we need to do some changes in the
code to make it more modular and adhere to software engineering best practices.
First of all, we need to enclose all this code in a function.
Let's call the function get playlist ID and in Python to define a function you can use this syntax.
Def and the function id which we just set will name it as such.
Next, we need to enclose the code inside the function in a try except clause.
This is a best practice, and you do this to ensure that the function can gracefully handle potential
issues and provide meaningful error messages instead of crashing, and you don't know what caused the
issue.
Again, I have to indent the code.
Here we rewrite the.
Except for now when working with the requests module, the docs and their errors and exceptions, which
will be in the appendix of this section.
And this is what you can see here.
It says we should use the response the trace for status.
To capture HTTP errors.
And also that all exceptions inherit from request dot exceptions dot request exception.
So we will use these two pieces of information in our code.
Knowing this let's incorporate the final changes to the function.
Let's go back to the code.
Let's first start by replacing.
Actually we shall remove this print as we don't need it.
And we can instead write the response dot res for status.
These other prints we can comment out for now because we will need them.
Ultimately we want to also return the playlist ID, so let's do that.
Return here FM stick playlist ID and we return the channel playlist ID.
Finally, in the except clause, let's write what we just saw the requests documentation.
So we write accept requests, dot exceptions, dot request exception as in.
And then we raise the exception here.
What we are saying is that any exception relating to requests will be set to the variable E.
And then we need to raise the exception.
So we write raise e.
Now if we modify the function we need to ensure that it still runs as expected to execute a function
in Python.
A popular approach is to use the if underscore name equals double underscore main block, which I will
write.
Now if double underscore name is equal to me.
Then we write the function name here.
If you have coded in Python before, you will have seen this.
And if you have to go over what it means.
This name part is a special built in variable in Python, and it gets its value depending on how the
script is run.
And this main part over here is the value of name when the script is run directly and not imported as
a module.
So what we are saying here is if we run the script directly, meaning we push the run script button
here in VS code.
When this happens, name variable is set to main.
The if conditional then becomes true and the contents of this block is run.
In this case, the function will be executed.
If, however, we have to run this script from another script, the name variable will not be set to
main, but it will be set to the name of the Python script, which in our case is video underscore stats.
Let's do a quick example and write the following in the video script.
First, let me close this here.
Let's add a print statement in the if block to showcase what happens when this script is run directly.
And then also we need to create another block to print text, highlighting that the script was executed
from another script.
In this case, the function won't be executed.
So if we were to run a directory, the if clause would get executed.
Let's do that.
And as you can see, the print statement of the if block got executed.
If, however, we were to run this main script from another script, let's call it import videos stats.
So import videos stats.py.
The S block will be executed and the function we just created won't run.
So let's first import the main script and then write a simple print statement showcasing where we are
running the script from.
Again, we're doing this to highlight that we are running this script from the import video underscore
setup UI.
Let's run the script.
And as you can see, the prints from the code executed and the function was not run like this.
We have gone over this famous double underscore name equals double underscore main.
If you didn't get this last part of the lecture that's okay.
Feel free to rewind and listen again.
It definitely won't come in the way of completing this project.
It is just a good to know and helps with code, modularity, and even for testing purposes.
Before we start, let's do a bit of cleaning.
Let's delete the JSON file we created to showcase JSON crack so we can press delete.
And also we can remove the prints from the if name equals main block.
That and also remove the block.
One final thing we can remove is the import video stats, as we will not be needing it in our course.
And actually, one thing that we need to do before closing off is to confirm that the script still runs.
Returning the variable.
What we see here won't output the result in the terminal.
So what we can do is uncomment the print.
We need to correct this variable name.
And now we can press run.
And there we have it.
The channel playlist ID is printed in the terminal.
And so we can confirm that the script is still working as expected.
Now let's comment the print again.
We save.
Let's stop here for this lecture.
In the next lecture we will push our code to GitHub.
But first we will need to introduce the dot env file.
I will see you then.