Similar to the first job.
The second job will only need to run if certain files within our codebase change.
Here we will also introduce the concept of job dependencies.
So let's follow the same structure as the first job and give this job an appropriate name such as the
following.
Also, same as in the previous job, we need to select the runner on which to run this job, which will
also be the latest ubuntu image.
So nothing new so far.
But now we introduce the concept of dependencies, which in GitHub actions you specify using the needs
command.
We would like it before we run the tests.
We have the latest updated Docker image.
So we need to ensure that the first job, which is the build and push image, runs before this job,
which we are defining right now.
So we specify the needs as we said, and also to specify the job name so we can copy this from here
and paste it here.
As we have said already.
The docker compose at the moment uses the env which we have saved locally.
However, this is not committed to GitHub for obvious reasons.
With this GitHub workflow, we need to find a way to specify the environment variables defined in env
inside this YAML workflow file, and we do this by specifying the env parameter, which references the
GitHub actions, secrets, and variables for all the variables in Docker compose.
If I had to list out all of the variables and secrets that we have in the env, this will look as follows.
By my count, I have 24 secrets and variables.
Notice how sensitive variables are defined as secrets and non-sensitive as variables.
These secrets and variables we specify in YAML file needs to be found in the GitHub actions as either
secrets or variables.
So we have to go back to GitHub actions.
Make sure that the secrets and variables that we have just defined in YAML file align with what you
have in the GitHub actions, secrets, and variables.
What this means for the Docker compose is that we need to remove these env underscore file parameter
references, since these variables will not be provided by env at this stage.
So we can comment out the code in the airflow command and also in the Postgres section.
This takes care of the airflow command.
Now let's go down to the Postgres service and also comment it out for the Postgres container.
An additional change that is needed is to include the following in the environment section.
So let's uncomment these out.
This is done since the database Initialization script needs these variables, and the GitHub actions
doesn't automatically load environment variables into the compose, so we need to ensure the variables
are parsed inside the Postgres service itself.
So like that we have done the required changes in docker compose.
Going back to the YAML file, as we introduce the job dependencies and environment variables, we can.
Same as before, write the steps of the job starting with the code checkout from the repository.
So let's go up and copy this code.
Having checked out the code, we now define the files that will trigger this job.
Should there be a change in them?
And for this step, it makes sense that we define any file that is inside the airflow DAGs folder.
The include folder which holds the configs for data quality tests.
And finally the docker compose file.
We don't need to define docker file and requirements.txt because before running this job, the build
and push image job will always be triggered and run before this job.
This is handled by the defined here for the needs.
Putting this all together, we can simply copy and paste what we have here.
We need to do a change for the ID since this needs to be unique in our case.
This was for the build step, but now we are working with the tests.
So let's change this to change files tests.
The action will be the same, and the files that we will be checking for any changes will be under the
DAGs folder.
And finally the docker compose YAML.
Now that we have our setup, we need to think how the jobs will be executed in order to continue writing
the YAML file.
If we were to do this manually, we would first.
Docker compose up to bring up all the services related to airflow.
Once we have the containers running, we need for the CI part to run the Pytest command in order to
run all the tests inside this folder, and for the CD part, we would run the airflow DAGs test command
for each Dag that we have in the Main.py inside the DAGs folder.
Finally, after we have concluded all our tests, we simply tear down the docker compose to clean up
the resources.
So if we had to write these steps that we just mentioned in the YAML file, making sure that at each
step we specify that we only run if any of the files we defined in this job were changed.
This would be as follows.
So first we define the docker compose up.
Taking care of the indentation.
Next we run the unit and integration tests.
And lastly, we define the end to end step in our main.py.
In the DAGs folder we have three main DAGs with the IDs produce underscore JSON, update, underscore
db and data quality.
So we can either define them explicitly here in the list, or we could generate this list dynamically.
In the production scenario it would make sense to use the latter.
However, we are only dealing with three DAGs.
And to keep things simple, I will go with the first option of defining the Dag IDs explicitly.
After we have defined a list with IDs for each Dag, we need to run the airflow DAGs test command in
an iterative way, making sure that we first start from produce underscore JSON, then update underscore
db, and finally the data underscore quality dag id.
To do this, we can define a for loop.
And if we were to write this down, it would be as follows.
As a quick side note, this add syntax over here ensures that each item in the array is treated as a
separate argument, while the command itself.
We have already seen in the end to end test section.
Same goes for the command to specify it here for the unit and integration tests.
Finally, we simply tear down the docker compose after we have run our dags.
This concludes the GitHub workflow YAML file.
What is left is to test it out.
The question is how we can trigger this workflow.
Remember that we designed this GitHub workflow so that we can trigger it in three different ways.
And this we did it in the on section of the YAML file which you see over here.
Now since we have done changes to the docker compose file, this will trigger the second job in our
pipeline.
The first job will also be triggered since we are using this needs parameter, but the actual job won't
be run since nothing in the requirements.txt or docker file have changed since the last commit.
So to also test out the first job, we will change the requirements.txt to use a more recent version
of pytest.
Let's say version 8.3.3 and we save like that.
Both the docker compose and requirements.txt have changes inside of them.
So let's add the YAML file and commit the new changes.
Finally we push.
So once we push the changes, if we go on GitHub we will see the execution of the pipeline.
Here you can see it over here and it's in pending state.
Click on details.
And here we are seeing the actual execution of the pipeline.
Hopefully it will give a success, but if not, we will look at the logs and see where errors might
pop up.
So it seems that the build and push image job is successful as we have a green tick, and now we will
wait for the unit and integration with also the end to end tests.
So we are seeing that the tests are passing.
Remember these we executed using this command by test.
Here we are executing the end to end Dag as well.
And we are returning also the logs from the video IDs.
So at face value it seems that boats jobs have succeeded.
If we hadn't, we would get a red X, so we had to go briefly in the jobs themselves.
You can see that the for example, the build and push image.
Here we have successfully logged in to Docker Hub.
We pushed the new image with the latest tag and also the commit hash.
This is some cleaning for build x, also some other post checkout code and the job is complete.
One verifications we can do is to go on Docker Hub and check the image.
Now has the latest tag with a new commit hash.
So signing into Docker Hub, the first thing that I can see is that we have a push which was only five
minutes ago.
Let's click on this.
And as you can see we have the latest image which is the following.
And as you can see we have a history of the tags that we had.
So first we had the initial 1.0.0 and also the point one where we did a small change.
And now we have the same image but it has two different tags.
It has the latest tag and also the commit hash which you see over here.
And to link things we go on git log we will see the latest commit will have this hash.
So as you can see it starts with one to 8 to 9.
Let's go and write the git log command.
And as you can see the latest commit also has this commit hash.
The 1 to 8 to nine.
And then it also has the commit message that we just committed.
Going also on the unit and integration and end to end tests, we can see the setup procedure.
Here we are running the unit tests integration tests which we saw while it was running.
But here we can verify that all the tests have passed the end to end tag tests as well.
These are all the logs that we get from airflow.
And here we also have or what we are seeing here the data quality tests.
Here you are also seeing that this workflow has a workflow dispatch events trigger.
We will soon touch upon this when we go into the manual triggering.