Now that you have everything set up and ready to go, I will explain the project outline in terms of
milestones.
The first milestone is getting the data.
In this course, we will be getting the data from YouTube and in order to extract this data, we will
be using the YouTube API.
The data will be using is from a popular channel which is that of Mr. Beast.
I think most of you have heard of him.
If not, no issue at all.
Who he is is relevant to what data engineers.
We are just interested in the data which we will clean and test to make sure it is of good quality and
loads to a storage system from where our colleagues downstream of us, like data analysts or data scientists,
can use for their analysis.
In our project, we will be using Python to build the scripts to interact with the YouTube API.
The second milestone relates to the loading and transformations part transformations, meaning we will
apply changes to the data to either make it more readable, or extract just a portion of the information
from the raw extracted data.
Loading implies saving the data to a storage system, which is many times called the data warehouse.
There are other variations of storage system names like Data Lake or more recently, Data Lake House.
Don't be concerned about these different names as they all serve the same purpose.
Just know that in our case, we will be creating a data warehouse.
The first and second milestone combined are what is known as the ETL or more accurately in our case,
ELT.
The choice between ELT or ETL depends on the order of loading and transformations.
In our case, we will first load the data to the data warehouse and from there apply transformations.
So in summary we will be performing ELT.
Traditionally, the ETL extract, transform and load process was used.
So the data once extracted would be transformed and then loaded into the data warehouse.
However, in the past years there has been a shift from ETL to ELT.
I won't get too much in detail about this, but just know that this change is driven by advancements
in technology.
Modern data storage solutions can handle large volumes of raw data efficiently, making ELT more scalable
and flexible than ETL.
In our case, we will be using Postgres as our data warehouse, and we will use Python to apply the
transformations.
Once we have transformed data loaded into our warehouse, we can move on to our third milestone, which
is to perform what are called data quality checks.
Data quality tests are essential to ensure that the data given to the downstream users, like data analysts
and data scientists, is accurate and reliable.
The most common data quality issues I can think of are missing data duplicates and incorrect formats.
As data engineers, it is imperative that you do these checks so the downstream users are confident
that they are receiving reliable and accurate data, which will be used for analysis and decision making
of the business.
In our case, we will be using a tool called soda.
The fourth milestone is then the functional and end to end testing.
Testing is a software development best practice, and is a skill set that will set you apart from other
data engineers.
For now, just know that we will go over unit tests, integration tests and end to end tests.
The details of these tests will be explained in the specific section and for testing we will be using
a combination of tools.
We will use two frameworks called Pytest and Unit test for unit and integration testing and Airflow's
testing features for the end to end tests.
The fifth and final milestone is implementation of continuous integration and continuous deployment
or in short CI CD.
For ci CD we will touch on many topics like deployment using Docker and automated testing.
For now, just know that ci CD is important because it helps keep your data pipelines running smoothly
when you apply changes to your code base.
We will discuss this in more detail when we get to this section.
And in terms of tools for CI, CD, we will use GitHub actions.
So once you conclude this final fifth milestone we will be ready.
Okay, that's all for now and I'll see you in the next lecture.