Airflow is made up of multiple components.
This diagram here shows the main components that make up airflow.
Let's start by first going over the airflow scheduler, which is one of the more important components.
The scheduler determines what tasks need to be run and when they need to be run.
Within the scheduler, there is also the executor, which determines where and how tasks are run.
There are a handful of executors in airflow, with the more known being the local sequential celery
and Kubernetes executors.
They have different mechanisms of how they work and are used in different environments.
For example, the sequential executor deals with simple single task executions while the executor runs
parallel tasks on a single machine.
So these two executors, we would use more for testing or development purposes.
Then there are the executors, like the celery or Kubernetes, that would be better suited for production
scenarios where distributed tasks are common.
In our case, we will stick with the default executor of the official airflow docker compose file,
which is the executor.
We will go over this docker compose file.
In a later lecture of this section, the executor will then direct the workers to execute the tasks.
In our case, we will only define one worker, but we're in a production environment where you are working
at scale.
You will probably need to increase the number of workers.
The airflow web server is the airflow UI where we can monitor and execute our DAGs.
In our case, this will be available at localhost on port 8080.
As a quick side note, I just mentioned the word dag, which you can also see in the image.
The word dag stands for directed acyclic graph, and we can split this three word definition as follows.
It is directed because the airflow tasks have a specific order.
Acyclic because there are no cycles, meaning a task cannot depend on itself.
And the graph refers to the fact that you can visually represent the tasks and their dependencies.
We will see how to build an airflow Dag in the coming lectures, but for now, let's come back to the
airflow components.
In this diagram you also see the metadata database.
This is where all the DAGs, code, scheduler, metadata, logs, variables, connections and much more.
Are stored.
Usually this is a Postgres database, but it can also be MySQL or for development purposes, even SQL
Lite.
We have gone over the core components, but there are some other, let's call them situational components
that we should discuss.
The first one being the airflow trigger.
The airflow trigger supports the variable operators.
An example of when this component would be used is if you have a sensor that waits for a certain file
to be uploaded to an S3 bucket, the trigger monitor is the S3 bucket until the file is uploaded.
Once the file is detected, the trigger signals the scheduler to resume any processing task post landing
in the bucket.
In our case, we won't be using the variable operators, so in the docker compose we will switch it
off.
Another situational component, which in our case we will use is a message broker.
By default this is.
Read this and you can see it here in this queue part of the diagram.
This is needed by the executor which is the one we will use.
And its role is to forward messages from the scheduler to the worker.
That's it in terms of airflow architecture.
In the next lecture we will go over the airflow directories.