End to end testing tests, in our case, the entire ELT route.
From the moment we extract the data from the API to where we are applying the data quality checks to
do an end to end test for a Dag, you can use the airflow Dag test command.
If we had to use the dash dash option for this command, we would see that this will execute one single
diagram.
All you need to specify is the dag id, so we can run the command using one of the Dag IDs that we have.
The first one being the produce underscore JSON and the python following the execution.
Here we can also see it the airflow UI.
So let's first sign in.
Let's hit the execute button from here.
So you are seeing that this Dag is actually running.
You can also see it from here if we refresh.
So it seems from here the Dag has finished running.
Let's go back to the UI.
And as we expect, the Dag has been successfully run.
What this means is that we will have a new JSON file in the data directory.
You can do the same with the other two DAGs just by replacing the dag id.
So the first one, or should I say the second Dag is the update db.
Here it seems that everything has run successfully.
Let's confirm by going on the UI.
And in fact, it did successfully run for both the staging and the core layer.
And finally we can run the data quality checks.
So I guess we have successfully run all the DAGs that we have created.
A question that you might have is would it be ideal if all this testing we did in this section could
be automated and triggered?
Once we do a change in our code base, this process of automatic testing should be included in a continuous
integration and continuous deployment workflow or for short CI, CD.
And this will be our focus in our last section of this course.