Now that we have all the functions to extract the data using the API, we need to think about how we
will load data into the data warehouse.
In truth, there are various approaches depending on your data architecture design.
You could gather all the raw data stored as CSV or JSON or in any other format, and upload these to
a data lake bucket in a string.
You could also decide not to save your data in a raw format, and push directly the data to your data
warehouse, which is a common theme in streaming where you need immediate access to the data.
In our case, considering the volume of data which is a few hundred rows.
And to keep things simple, we will use the first approach.
But instead of storing in the cloud or some data lake, which is what you would do in real life, we
will simply store to a part on our local machine.
To do so, we just need to create a function that takes the data we just extracted and save it as a
JSON at a specific file path on our machine.
So let's define a function.
Let's call it save to JSON.
So the save to JSON, which takes as input the output data from the previous function.
So extracted data.
Now we need to have a file path where we will save the JSON files.
This can be in our current directory under a folder.
And let's call it data.
So let's create a new folder and name it data.
Since we will run the ELT pipeline once a day, we can also specify in the JSON file name the day the
file was generated.
For this, we first need to import datetime module at the top of the script.
So this would look as follows from DateTime import date.
Now going back down.
We define the file paths.
So file path equals start an f string.
We're taking the current directory using the data folder we just created.
And the file name will have the structure of YouTube data.
Taking into account the day that the script is run and it will be of type JSON.
We can now use what is known as a context manager to write the file.
Context managers are an efficient way of opening, writing and closing files as it takes care of resource
management.
The most popular context manager, which is what we'll be using, is the with statement.
So we define the with open which needs the path where we will store the file and the action we want
to do.
In our case we want to do a write.
So we would set the mode to W.
If we wanted to read the file we would set the mode to I.
So let's write what we just said in the with statement.
With open we set the file path as the first input.
We said we want to write.
So we use W and we also will specify the encoding which will set it to encoding equals UTF dash eight,
which simply ensures that the file can handle special characters.
Then, since we need to save data as JSON, we used JSON dump which takes as arguments the Python data
represented by the variable extracted data.
We start tracking this down so JSON to dump.
We said we will take the extracted data.
Also, we will take the file objects to which the JSON data will be written.
This is represented by what we just defined the JSON underscore file.
We can also define the data indent, which is usually set to four spaces as it makes the output pretty
printed and therefore easier to read.
Let's add the indent to four.
And lastly, we also include the argument ensure asking equals false which allows JSON to include non-ascii
characters.
What we have now in this video underscore stats.
Python script is the whole extract part of our pipeline.
If we were to run it, we would end up with the JSON file for today's date under the data directory.
Before we get the file, let's do one final change at the bottom of the scripts.
And the double underscore name equals double underscore main parts.
So the change is to simply call the function we just created.
And we do that here calling the video data variable.
So let's press run.
And we should ultimately end up with the final JSON containing the video variable values for this day
at the specific time.
This might take a while since we're looping through all the videos, but we should ultimately see the
JSON file.
And there's this data directory over here.
And as you can see, git picked up a change.
And if we were to open the JSON file for today's date, we will see the contents of all the videos that
are present on Mr. B's YouTube channel.
So that's it for this section.
We finally have the main scripts that will extract the data from YouTube using the YouTube API.
And in the next lectures we will go over how Docker will play an important role in our architecture.
I will see you then.