Going over to the main.py file, where we have already that that produces the JSON file with the raw
data.
We can copy the Dag definition and update it for our needs.
However, before we copy, we just need to import the main functions as modules for the data warehouse
scripts.
And now we can copy this drag and edit it.
For our purposes here we will update the Dag ID and we can update it to update underscore db.
We can also change the description to something like Dag to process JSON file and insert data into both
staging and schemas.
And we should also change the schedule to a time after the first runs.
So let's say 3 p.m..
We then just define the tasks for the update of the staging and core career tables, so this would look
like this.
Update staging will take from the staging table function and update core will take on the core table
function.
These we can remove.
And finally we set the dependencies so that the update staging task runs before the update core task.
So the actual Dag is written for the data warehouse part of this course.
What we need to do now is to actually spin up the Docker containers and see if this Dag runs, or there
are some errors that we have to eliminate.
So let's first run the docker compose up minus d command.
Let's just do a docker PS to make sure that the airflow containers are up and running.
So the containers seem to be running.
And now we can go on the airflow UI, which we can access through the localhost on port 8080.
Let's insert the credentials which we know we can get from the env.
And there seems to be an error with the import.
Now I realize that I might have misspelled the Psycopg2 import.
So let's go back to the data utils and let's correct this.
Let's refresh.
And we finally have the update underscore db back.
As you can see here I'm not sure if you notice the produce underscore JSON just ran.
This is because we just spun up the Docker containers.
And the Dag from the last run was run now.
So let's go on the update and unpause the Dag.
As you can see, we have an error if we go on the logs.
Let's see what the error could be.
I'm seeing here there is another spelling mistake.
So let me.
That's right now.
And let's rerun the Dag.
So from what I can see, we have an error when we come to insert the actual values.
I am looking at the actual script that we have.
And from what I can see, we have this extra bracket here which is causing the error.
Let me just see that we don't have any more of these errors here.
Seems to be okay as well.
And I think we are good to go.
Let's refresh and let's try to rerun this again.
So as you can see, inserts are happening into the staging table and it seems to have completed successfully,
but the core table appears to have an error.
So let's see what this error could be.
So what I'm thinking is is we have another spelling error.
Let me go inside the data warehouse UI script, trying to figure out where the error is coming from.
So it's not here.
Let's see if it's.
So here it is.
Here.
So now that we have corrected the spelling error, let me go back here and let's try and run this.
Let's do a refresh and run again.
Finally success in both the staging table and the core table.
We already had data in the staging table from the first success.
So now the actual logic that is kicking in is the update here.
We should have exactly the insert logic that is triggered, since there are no videos or no data inside
the table schema.
So that's it.
A bit of errors along the road, but that is normal when you are testing and developing your DAGs.
This is actually a good practice for me as well, to showcase how you would actually tackle solving
issues in a Dag.
As you can see, the logs section here is very important and one should always be looking at what the
actual error is telling us and where we might have the actual error.
So that's all I have to show you when it comes to running the Dag using airflow.
However, this leaves us with the question where and how can we see the data that we just inserted?
So this will be the next point of discussion in the next lecture.
I will see you then.