Now that we have a better understanding of the airflow architecture in terms of components, let's shift
our focus to the airflow directories.
What we will do with Docker is create a dark code on our laptops.
Then we will use what are known as Docker volumes to mount the files from our local directories to the
paths inside the containers.
Once we have these volumes set up, any changes we make in our local directories will be reflected in
the airflow Docker containers.
There are four common directories in airflow, with the two most important ones being the DAGs and the
logs folders.
Let me just get a notepad and write these folders names down so we are all on the same page.
So I mentioned the DAGs folders which we will use heavily, and also the logs in the Dag folder.
The Dag Python code will be written while the logs folder will contain logs from task executions and
the scheduler.
The other two directories are the config.
So let me write this down, which can be used to customize airflow, custom configuration and the plugins
folder where you can put your custom plugins such as custom operators, sensors, hooks or other airflow
components.
Now in our case, we'll use the DAGs and Logs folders while the config and plugins we won't need to
use.
However, we will need to create some other directories.
One of these directories is a tests folder where we will store the code for the functional tests.
The data directory which we have already created will contain the data files, which we save in JSON
in the extract part of our pipeline, and we have already seen this file being generated in previous
section.
And finally we will also have an include folder.
Which is normally used for any additional files or resources that your DAGs might need.
These are usually SQL scripts, Jar files, SSL certificates, or any other resource, but in our case
they would be used for YAML files relating to the data quality tests.
We will discuss this YAML file when we get to the data quality section of the course.
And with that, we have covered the main theoretical aspects for the airflow architecture and directories.
In the next lecture we will touch again on the env file.
I will see you then.