We have already gone over the dot env file in the first section of this course, and why we need it
to handle sensitive variables.
Apart from sensitive variables, we also discussed how we can also store non-sensitive variables in
the env file that relate to specific environments setups.
For example, a production scenario will have different configuration settings when compared to a development
environment.
So you will have one env for dev and one for prod.
In this section we will build upon the env file we already have and use it to reference variables like
database connection credentials, airflow parameters and some other variables.
I have already made the necessary changes to the env file, as you can see here.
The DMV you see here is also available in the previous slide document.
Let's go over these environment variables and also tell you which ones you need to change and which
you can keep.
As you see here in terms of the environment variable definition names like Docker Hub namespace repository
PostgreSQL username.
It is imperative you don't change any of these names, as they will be referenced by other files like
Docker compose.
What you can and in some instances should change are the values of these variables, and we'll cover
this shortly.
The first set of variables relate to the Docker Hub credentials, which are these ones.
The namespace is your Docker Hub username, so make sure you change it to your username.
While Docker Hub repository is the name you gave to the repository.
In my case, I gave it the name YouTube underscore API underscore erd.
Moving on to database connection credentials.
As we have already mentioned, we will use Postgres for all our databases.
And in terms of databases we have three.
These being the first one is the metadata backend which will store the data related to the metadata
used by airflow.
The second is the database that will store the results of tasks executed by the executor.
And the final Postgres database is the database where we will store the data extracted from the YouTube
API.
So what you see here under Postgres connection is the connection parameters to Postgres instance that
will contain the above three databases.
And for this connection you will need the username, password, hostname and port which is 5432.
The default Postgres.
You can keep these values same as you see them here.
This is especially important for the hostname for the host.
It is important that you keep it as Postgres because the service name for the Postgres container will
also be Postgres.
In the docker compose file, if it's not, airflow will complain about host resolution for the three
different databases, which is these configurations here.
The variables specified are the database name, username and password.
I would recommend you keep these the same as I have them, especially for database names.
So it is clear which databases will store what.
If you decide to change any of these credentials, for example, the one of the passwords of database
connections, make sure that they do not contain special characters such as the dollar sign, the At
symbol, or any other special character.
When I was testing, I had a password that contained one of these special characters, and I was getting
an issue with parsing when I integrated these credentials in the Docker compose.
So again, feel free to use different passwords which you can create with, for example, online password
generators, but be cautious of these possible issues.
Again, I would recommend to use the same credentials that I have.
The remaining airflow parameters are the airflow Uuid or the user ID, which we set it to the default
of 50,000.
This means that the airflow containers will run with the user ID 50,000.
This step of adding the airflow Uuid into the env is also recommended to be done in the airflow documentation,
to avoid a warning that might give you relating to this fluid variable not being set.
The username and password you see here are the credentials you will use to access the Airflow web server
UI.
Feel free to keep these or change them as you wish.
The final airflow parameter is the net gain for Net.
Keys are used for encrypting and decrypting sensitive data stored in the metadata database.
The airflow docs have a code snippet of how we can create a foreign key, as shown in the course slides.
This is the piece of Python code that the airflow documentation recommends for creating a fairness key.
So this would require us to install the cryptography library using Pip and is the approach you should
take for production scenarios.
Since we are not in a production scenario and the passwords we create are not intended to store sensitive
information, there are other simpler ways of generating a key.
One of them is using a Fernet Key generator website.
In my case, I am using this website called Fernet Key Generator, which I will add to the appendix
of this section.
You can press Generate key and you can use this in your env.
If for some reason at the time you're viewing this course, the website is not working, you can either
use the Fernet Key in Miami.
You can also try searching for the website that does the same thing, or as a last resort, created
using the Python code as detailed in the docs.
Finally, the last two parameters in the env are the API key, which we are very familiar with at this
point.
Make sure that you use your own API key and that you don't share it with anyone.
My personal key won't work for you since I will have deactivated it by the time you are viewing this
course.
Here I also include the channel handle which we had seen was Mr. Beast.
And that's it for this lecture.
In these lectures, we have built the foundation that will help us understand how Docker compose will
be structured for our airflow setup.
I will push the docker file and requires a txt to GitHub, and I will see you in the next lecture where
we go over the docker compose file.