One further change that we would need to do before we start the CI CD section is to revisit again the
DAGs we built.
Currently, we have a problem with the logic of our DAGs, as we don't have a guarantee that they will
run in the order that we expect them to.
Sense scheduling is solely time based.
If the first Dag is set to run at 2 p.m., fails or is delayed, which is this one here, the second
Dag is set at 3 p.m., will still run and it will fail as well.
Same story for the third Dag that currently runs at 4 p.m..
This is also inefficient since if the Dag only takes five minutes to run, we have to wait another 55
minutes before the second Dag is run and another hour before the last Dag is also run.
A simple change that we can do is to join all the tasks in our three DAGs into one consolidated Dag.
This is a completely fair approach in terms of Dag architecture and would avoid the issue we currently
face.
However, I will leave the architecture as it is right now with three DAGs to go over the topic of Dag
dependencies, which is a challenge you will for sure face at the workplace.
So in our case, what we can do is to keep only the first Dag with Dag id produce underscore JSON to
still have a schedule trigger at 2 p.m., while the other two Dag IDs.
This being the update db and data quality DAGs will be triggered once the subsequent Dag finishes successfully.
And this can be done using the airflow trigger.
Dag run operator.
The trigger Dag an operator takes a minimum two arguments the Dag id to trigger, which is set using
this parameter here, and the task id which is inherited from the base operator.
So with the trigger diagram operator we have a chain of dependency between DAGs, which by default triggers
the subsequent Dag.
If all the tasks in the previous Dag have completed successfully.
If a task fails, then the trigger Dag operator won't run because the dependency chain is broken.
Therefore, the Dag will not be triggered.
This is exactly what we want since, for example, if the produce underscore JSON dag fails, we wouldn't
want the update underscore db to run as there will be no JSON file to update the data in the data warehouse.
In terms of changes, the domain of script, I have done the changes already and we can see the differences
by using git.
The very first change we need to do is to import the trigger Dag run operator.
This is done with this command over here.
Going down to the first Dag definition, we will keep the schedule for the first Dag at 2 p.m. instead
of reusing the same Dag variable.
We can now distinguish between the three dogs and the right deck underscore produce for the first Dag.
Dag underscore updates and Dag underscore quality.
All of these replacing the command Dag variable.
Going back up to the first Dag.
The biggest change will happen here where we introduce the first task using the trigger diagram.
Operator.
In this first Dag we are adding this additional task trigger underscore update underscore DP in the
task dependencies chain.
Its purpose will be to trigger the next Dag which is the update underscore db dag.
This we are defining by the trigger underscore underscore id parameter here.
And again to reiterate, we also remove the schedule for the last two DAGs by setting the schedule equal
to none for both Dag number two and number three.
Repeating this step for Dag two, we define a trigger underscore Underscore quality task which is this
one over here, which is set to run after the update underscore core task.
And like that we also have a dependency between Hdac2 and Hdac3.
If we were to go back to the airflow UI, once we do a refresh, we can see that the schedules are now
conforming to the changes that we just did.
Now, if we had to manually trigger the first Dag in the sequence, we would see the others being triggered
automatically.
So let's do that right now.
Let's trigger the first Dag in the sequence.
Here we are seeing that it is currently running.
Let's give it some time to actually run and let's go back here.
So we will see that the update underscore DB should be triggered once the first Dag finishes.
As you can see the producer Jason has successfully finished and updated be was run same of course for
data quality.
You can see that once this was run, the second Dag data quality was also executed.
So we have to go update underscore db.
We will see that this just triggered right now.
And even data underscore quality.
Now that we did this change to the DAGs script, we also can't forget to update the unit test script
here.
We no longer have four tasks but now we have five.
And same goes for update underscore db.
We now have three tasks.
The data quality Dag was untouched in terms of number of tasks.
So that will stay true.
Now that this change has been applied, we can commit and push to GitHub so we can finally start the
ci CD section.
I will see you then.