-: Hi everyone, and welcome back.
In this lecture, we're gonna get our environment set up
and I want to introduce this more high level overview
for those of you that wanna try and solve it on your own.
And this may look familiar,
in the last lecture you saw this,
so if you read through it, my apologies,
we're just gonna go through it really quickly.
First things first, in this project,
we're really aiming to keep it as simple as possible
in the sense we don't need to import too many libraries.
We're basically just gonna be using NumPy.
For that, we just need to import NumPy as NP,
usually the common reference for NumPy.
And we also want to set up our environment.
As you'll see here,
and we're just gonna go through this again
really quickly as an overview,
the first step for our Q-learning
is we wanna define an environment
that the postman has to navigate,
we need that environment set up
so we can actually iterate and go through it.
In this lecture,
the environment's going to consist
of states, actions, and rewards.
States and actions are inputs for the Q-learning AI agent,
while the possible actions are the AI agents' outputs.
Our states, we can think about
and look at this image as our representation,
the states in our environment
are all the possible locations within the city.
We can call this "example city".
Some of these locations are the city boundaries
which will be our black squares,
while other locations are aisles
that the postman can use to travel through the city,
those are gonna be the white squares.
The green square indicates
the item, packaging and shipping area.
The black and green squares
are what we're gonna call terminal states.
So overall, our goal or our AI agent's goal,
we wanna use the shortest path,
we want our agent to learn the shortest path
between the item packaging area, which are green,
and all the other locations in the city
where the postman is allowed to travel.
In the above image,
we have 121 possible states or locations within the city.
These states are arranged in a grid 11 by 11,
each location can hence be identified
by its row and column index.
So what would be our first step?
And this is where I want you guys
to start thinking about how you can define it.
We need to define our environment.
This is a good example of our image
and how we're gonna approach it,
so how would you model that?
Remember, we're using NumPy.
So we need to define these boundaries
and we can define a 3D NumPy array
to hold our current Q values for each state and action pair.
As we see our representation,
and for those of you who are not familiar with it
or maybe this is new or you want to just have a refresher,
the AI A-Z handbook from this course
is extremely helpful, highly recommended.
So what do we have to do here?
We can actually,
let me just expand this really quickly for us
so we can view it a little easier,
let me just add some code cells.
We're gonna define our 3D NumPy array.
How would you go about this?
So we have some options,
but the most really straightforward and simple option,
let's call it environment rows,
rows, and let's set it to 11, it's an 11 by 11.
Then we can also do an environment_columns,
and we can also set this to 11.
Lastly, we can set our Q values
since we need to add our NumPy
with the environment rows, environment columns.
And we can set this as Q values equal to NumPy.zeros,
and we need to use our environment rows,
environment columns,
and we have our 3D NumPy array,
our 3D environment representation set with our environment.
Awesome.
Now, and we're gonna leave it off here
but I want you guys to start thinking about
how to solve this
since you already have your environment set up.
The next thing that you're going to want to do
as a hint is set up your actions.
Your agent needs to be able to move through the environment.
So how would you represent that?
How would you write that in for this problem?
Let's leave it off here.
In the next video, we are going to revisit those actions.
Awesome, I'll see you guys in the next video.