In order to better understand the steps needed, I have listed them in point format, so we are all
clear on what needs to be done with regards to the first point.
The tables are what will make up data warehouse.
In order to group these tables, we will define schemas which are nothing more than a way to logically
group objects.
In our case, these objects would be mostly tables, but you can have other objects like views, stored
procedures and more.
Moving on to the second point.
When you create the tables, you will have no data.
So we need to define functions that will insert, update and if needed, delete data from these tables.
To do this we will create functions in Python.
Finally, the third point we need a way to transform the raw data that we just inserted into the raw
layer into the refined layer where we will have our refined data.
For this, we will also use Python.
If we had to represent the data flow from data source to the refined layer of the data warehouse, it
would look like this.
We have the data source, which in our case is the API.
We extract and load the data into the raw layer of our Postgres data warehouse.
The data in this layer will be under the staging schema.
Then we perform transformations to get the data from raw to the refined layer.
The refined layer will be represented by the core schema.
In reality, a further refined layer you will also have what is known as the data mart or presentation
layer.
The way you structure this data mart layer is company specific, but you could have, for example,
a data mart for each department like marketing, sales and finance.
In this course, we will put our focus only up until the refined layer.
That's it for now and I will see you next to set up the connection to the data warehouse.