Now that we have the tables set up, we need to populate them with the API data that we have saved as
JSON files under the data directory.
So let's create a data loading script.
Let's do this under data warehouse data loading dot pi.
And let's create a function that will be responsible for opening the JSON, reading the JSON data,
and parsing it into a Python object.
So let's create the function named load path.
And before we actually start coding this up, we need to do some imports.
First one is import JSON from the time we need to import date.
And finally we need to import logging.
Now the JSON data modules we have already used in previous lectures.
However, the logging python library we have not.
I will leave the link to the logging documentation in the appendix of the section now The logging module
docks.
Once we have imported logging, we can define a logger object and this will look as follows.
So the logger object will take from logging logger and its name.
This is the preferred way of handling logging in your scripts, and it is a best practice to use this
module and not the Python print function.
In my personal experience, I have found that I use print for troubleshooting and debugging, while
I use logger in production code to output logs from Python functions.
So if we are to continue building the Loaddata function, first we specify the file path to point to
the data directory where we save the JSON in the extraction section of this course.
So the file path would be.
Next we can open a try except block and lock the action which we are performing using Logger.info.
In this particular case, we are processing the file so we can write the following.
If you underscore data and we define today's date.
Same as when we save the JSON, we use the width context manager.
And in this instance we want to get the data.
So we open the file in read mode using the R mode as we will see here.
And then going with the UTF eight.
Next we define the data variable that will contain the JSON data of the API.
This you can simply define this data and return it.
It's.
One thing that I will point out here is that when we define the variable data, we are effectively loading
the entire contents of the JSON file into memory.
If the JSON file is small, like in our case, this won't be an issue.
However, if the file is in the order of gigabytes, then we will have performance issues.
And if the system you are running on doesn't have enough memory, you can run into out-of-memory errors.
Possible solutions for this would be to stream the JSON using a library like JSON or JSON file, line
by line.
Now coming back to the code, if any of the code in the try statement fails, we use the accept clause
and in our case, possible failure could be due to either the file being found or the JSON data is invalid.
In either case, an error will be logged and raised.
And we can use the logger dot But this is a nice feature of the logging module where we can log a message
with level error.
We can define the file path if there's an error.
We can also define the JSON data as invalid exception.
So now we have a function that will store the loaded data as a Python variable.
So before we wrap up here I'm just using two errors.
This t here needs to be capital.
Since if we look under the data directory the t is also capital.
So like that the file wouldn't have been found and we would have gotten the file not found error.
Similar.
We can also do an underscore here to keep the same file name standard.
In the next lecture we will look into the insert updates and deletes.