Now that we have covered the theory aspect of airflow, we can now go over docker compose file.
Docker compose is a YAML file that allows you to define and run multiple Docker containers with a single
command.
Airflow provides a docker compose file that you can customize for your needs.
Simply go to running Airflow Docker in the How to guides, scroll down to fetching Docker-compose.yml.
You can press here and this will be the docker compose file we will be using.
I will leave a link to this page in the appendix of the section going back to VSCode.
In our case, I have already created a docker compose file, which is the one to see here.
I copied the contents of the base of our compose and made some changes, which we will now go over.
As you can appreciate, this docker compose file is quite large, and although we will go through the
more important parts of it, I would still recommend that you go through it on your own time as this
will further your learning.
So to recap, in this lecture, I will only mention the most important aspects of the docker compose
or the parts where we have changed from the default file provided.
So going back up, we can start off by going through the airflow command part.
This airflow command relates to common variables that will be available to all airflow containers.
These airflow containers are the web server scheduler and the trigger.
And by default, don't include other airflow containers like the ones for Reddis or Postgres.
The first variable that we have here is the image, which is expecting a Docker image.
We have already built and pushed a Docker image in one of the previous lectures.
So here we will reference it.
One thing that you notice immediately is that we are not referencing the image directly, but are you
using the dollar sign curly bracket syntax?
Using the syntax, we can reference the environment variables from the env.
The advantages of this we have already covered.
But to reiterate, this makes the code more flexible and reusable, as you can use the same variables
in different environments like testing dev or prod and using EMV, you don't show sensitive variables
in your code.
Now we need to pause for a second.
Here we are referencing this image tag variable that is in the docker compose, but is not in the EMV
environment variable file.
So let's correct this right now.
Let's set the image tag equal to the image tag version of 1.0.0, which we had defined in an earlier
lecture.
Let's save this and let's go back to the docker compose file in order to use it as EMV.
We need to reference the EMV file under the env underscore file parameter.
This tells Docker compose to use the env, which is found in the same directory as docker compose.
If for some reason you don't have them in the same directory, you would need to change the relative
path of the DMV.
Moving on to the airflow.
Common environment variables for executor.
We have already mentioned this, but we are confirming here that we will be using this executor.
Now we have a number of connection URIs which we have already gone over in the lecture.
And here is where we apply the theory.
This URI for example is for the metadata database.
And this URI is for the backend which stores results executed by the workers.
In both instances we are referencing database connection parameters that we have a number of these.
You can see them here for example the metadata database username password and so on.
And the same thing for the backend user name, backend password and so on.
Here we also reference the fairness key from the DMV and going a bit down.
We also have some true or false settings which you can go over in your own time.
Scrolling further down, we get to the airflow connections and variable settings.
These variables don't come by default in the docker compose, so you need to add them as shown here.
The first one being the airflow connection to the database that will store the YouTube data API for
connections can be written as URIs using the syntax airflow Con, which is the first part here, and
then the connection ID creating the URI.
Since we will use Postgres, PostgreSQL will be the connection type and the rest of the URI will reference
the connection parameters for the database.
The other variables that we will need to introduce will be used in the Dag scripts, and are defined
using the airflow file underscore variable names syntax.
So here we have an example.
For the API key we have the airflow where and then we are referencing the variable name which in this
case is the API key.
Same goes for the channel handle.
Now these final five variables that you see here.
We don't need to define them at this point in time, but they will be needed for when we start working
on the integration and data quality tests.
So I will simply define them here for when they will be needed at a later stage.
Moving on.
We now go to the volumes parameter.
Before we explain what we are doing here, it would be better that we are all aligned on what volumes
are and why we use them.
So volumes are a powerful tool in Docker and they are primarily used for data persistence.
When a Docker container is stopped or removed, the data is stored.
We will see this in action when we go to the Postgres service.
Volumes, however, are also used for mounting, which is what we see here.
These directories went over them in the theory lecture and what files they will contain.
Here we are specifying that we want the contents in our local directories, which is on the left hand
side, to be mapped to inside Docker containers, which is on the right hand side.
So let's take for example the content that is under the data folder.
We are referencing this line 97.
Here we are saying it's on the left hand side our local data directory, which contains the ELT data
from the YouTube API.
We want it to go inside the Docker containers on the part that you see here on the right hand side.
This is what mounting is in the context of Docker, and you can imagine how powerful this feature is
that covers most of the changes we made in the airflow coming part of the compose.
So now we can move on to the actual services which translate to the containers that will be spun up.
If we go further down, you will see the services that we will introduce.
And the first one is Postgres.
Here we are specifying a number of things.
We are saying the container name should be called Postgres.
If not, it will assign a default name that will be composed of the directory you are in, which in
my case is YouTube, ELT and the service name, which is also Postgres.
We are using the Postgres version 13 and we specify the file under the env underscore file parameter.
This will do it since the Postgres container is not a part of the airflow command containers, so we
need to explicitly define it here.
For environment variables, we also specify the root user and password for Postgres connection.
Now for ports here we have two ports.
Both are 5432.
The 5422 on the left hand side is the port on your laptop, while five four, three two on the right
hand side is the port inside the Docker container.
So here we are mapping the laptop port to the Docker port.
Both will be 5432 for Postgres, which is the default.
Now for volumes.
We have already described what they are and we already saw how they work for mounting local directories
to directories inside Docker container.
What we haven't seen is how volumes are used for data persistence.
And in this example we will get to see both the persistence and also mounting.
The first volume we have defined here ensures data persistence through a named volume, meaning that
a volume of the volume, which is this one.
Here we'll be creating on our laptops that will contain all the data related to the Postgres databases
we have, be it the metadata, data related to the celery executions or the data from the YouTube API.
And this data will be stored in the var directory in the Postgres container.
By defining this named volume, we therefore ensure that the Postgres data will be saved even if the
Docker containers are dropped.
The second volume, which you see over here is referencing an initialization shell script.
This will be used to create the three users and associated three Postgres databases.
So what this volume is actually doing, it is mounting this shell script over here from our laptop inside
the Docker container on this part on the right hand side.
So let me pull up the shell script which is found here, and the script I have already created.
And we won't spend too much time on it, but we can go over it briefly so we are aligned on what it
does.
So the first line of the script is known as the shebang.
And what we are effectively saying is that we will use the bash shell to run the script.
The set minus flag exits on any error, and the set minus u treats unset variables as errors.
Next we are defining a function creates user and database that takes three arguments the database name,
the username, and the password.
Then it uses SQL to create a user and database, granting all privileges to the user.
As a side note, psql is the command line tool used to interact with Postgres databases.
Here we are also seeing the function is called three times for each individual database that we have.
And ultimately we are sending a success message.
These messages you can see in the echo commands as well inside the function and outside the function.
A final note on this initialization script is that we could have created the three databases using a
different approach.
We could have initialized three separate containers with each database having a dedicated container.
So we had to go back to the docker compose.
We would have created three Postgres containers and each container would have one instance of the database
respectively.
This would result in very good isolation as all the databases are separated in different containers.
However, it is resource intensive and for small lows like in our case, there is no need for it.
Before we move on to the health check, I also wanted to mention that the named volume that you see
here would also be referenced in the end part of the docker compose, which you see here.
Now scrolling back up, the final part of the service is the health check and the restart policy for
the health check.
We are monitoring the health of the container to ensure it's running correctly.
In this case, we are testing the connection to the metadata database.
This is what you're seeing here.
And for the restart policy which is this part over here?
We are saying that if the container stops, it will always be restarted automatically for the other
services in the docker compose.
We can keep most of the default settings.
Just be aware that we have read this, which is the message broker used by celery executor.
And this function is to forward messages from the scheduler to the worker.
We also have the web server, which is the UI that we will be able to access using this URL on port
8080.
Going down we also have the scheduler, which as we know is the orchestrator that triggers tasks to
run based on our schedule.
And the worker that executes these tasks.
As a side note, here we are defining one worker.
But you could have multiple workers depending on how many tasks you need to run concurrently.
In terms of Docker compose, you can increase the number of workers by defining another container.
So for example, if you wanted to have two workers, you could have one worker service named airflow
worker one won with these properties and the other would be airflow.
Worker two with the same or slightly different properties.
Moving on to the trigger air.
As we have said in the previous lecture, we don't have the variable tasks, so we can switch it off
for our use case.
And as you can see, it is commented out.
Now the airflow init container only runs at the start, and it is an initialization container which
ensures the environment is properly initialized and set up for the airflow containers.
If we go a bit down, we will see it in the environment.
Here we are specifying two environment variables.
These are the username and the password that we will use to access the airflow UI.
The airflow CLI container designed to provide a command line interface for interacting with airflow,
which will be useful for running airflow commands directly from within the Docker environment.
Now the flower service references the flower, which is a monitoring tool for celery that gives insights
into task execution and resource management.
Feel free to explore this monitoring tool on your own time, but it will not be covered in this course,
so we will switch it off in post.
So like that, we have covered the docker compose that we will be using at a high level.
A piece of advice I can give you relating to named volumes is to be sure that before you set up the
containers, the variables and the credentials defined in the env are set and final.
Let's go to an example of why this is important.
Let's say you spin up Docker containers with a set of Postgres credentials using the docker compose
up command.
This will create, in our case the named volume and docker compose, which is this one Postgres db volume.
As a side note, we will be doing this exercise in the next section.
For now, I just want you to understand the scenario.
After we have spun up the containers we then want to stop Docker containers.
So we do docker compose down.
After doing that, for some reason we change the password for the metadata database, and then we tried
to spin up Docker containers again.
This will result in an error since the Postgres volume has the previous credentials stored.
The new windows credentials will conflict and the containers will not be spun up.
If you find yourself in this scenario for this course, I would recommend that you revert back to the
previous password or if you have the credentials and face this issue, delete the volume and spin up
Docker containers again.
Deleting the volume implies you will lose all the data stored in all the databases to delete volumes.
You can do this by either going on Docker Desktop application.
In the volume section, find the volume in question and click delete.
So let's do that right now.
So this is the Docker desktop.
And to look at your volumes you can go in the volume section here I have another volume.
This is from another project.
Just ignore it.
But in our case once we have actually spun up.
Okay, we will get the volume that you see here.
And it is a simple task of clicking here and pressing delete.
Now you can also do this deletion through the CLI.
First you run the commands to list docker volumes.
The command is docker volume list.
You will see the volume that I just showed you.
And then to actually delete the volume you would do docker volume rm and the volume name.
So in our case it would be.
And as I mentioned, you will only do this in the scenario where you're facing issues spinning up the
containers because you did a change in the credentials in the dot env.
So that's all that you need to know for docker compose.
I will see you in the next lecture, where we go over the spinning up of Docker containers and exploration
of the containers themselves.