-: Hi, everyone, and welcome back.
In the last lecture, you saw how we used
our helper functions to really set up the core
of our functionality to train our model.
So we had our terminal state, our starting location,
our next action, next location and shortest path.
Really defines the majority of how our agent
or how our environment's going to work.
The last thing we have to do is use our training.
In order to use our training,
it's pretty straightforward in the sense
of what we need to include,
but how we go about it is a different story.
That being said, also, if you guys use a different approach
or you want to customize this and experiment with it,
it's highly recommended.
In addition to that, if you use a different approach
and you wanna discuss it,
please feel free to share it in the Q and A,
more than happy to discuss it with you.
It's such a great idea,
because there's so many ways to solve this,
and you might find a way that's way better,
more advantageous.
We could look at pros and cons.
Again, more than happy to discuss it.
All right, so what do we have to do?
We have to launch our training.
For our training, we want to use the epsilon,
so basically the percentage that we're going to take
for the best action instead of a random action.
So let's set our epsilon,
and let's set it to 0.9,
and we're gonna do something similar with our values
for our discount factor and our learning rate.
So let's do discount factor equal to 0.9.
Let's also take our learning rate and set it to equal 0.9.
That's going to be the rate which our agent
is going to or should learn.
And we also want to take the number of training episodes.
Let's call it N training episodes.
So we have number of training episodes.
Let's set it to a thousand.
How many times this is going to train
or how many episodes this is going to train on.
Awesome, now essentially, what we want to do
is iterate through each episode.
We wanna take the range of our number of training episodes
and set, basically, our epsilon.
We want to find out our temporal difference.
To set that, we need to look at some previous Q values.
We need to calculate our starting position
and our action index.
So let's get started.
Let's take a big for loop.
So we have our for episode in the range
of our number of training episodes.
There we go, barring any syntax error on my part there.
We want to take the row index
and the column index for our grid for our maze column index
and set it to our get starting location.
We need a starting location,
so let's have that for our each episode.
Now, we can set our terminal state.
While not is terminal state.
We wanna look at the row, index, and the column index.
Okay, next, let's set our action index for our next action.
So we do need our next action as this agent
iterates through the maze.
For our action index, we can use our get next action,
and we can use our row index column index,
and let's call our epsilon.
So we want to have this, instead of our random action,
let's use column index and we need epsilon, awesome.
If I could spell correctly here, we want epsilon
All right, let me just delete this.
Okay, let me grab the rest of the code,
and we'll walk through it so we don't have
to watch me typing out each line,
but just keep that in mind how we want
to approach this of setting the old row.
We would need an old row index.
We would also want to have the rewards
for our columns and our rows.
We need to take a look at Q values and old Q values,
and we need to calculate our temporal difference.
Okay, so you see the change in code
or the update to the code.
We are taking our old row index
and our old column index to look
at our row index and column index,
our row index column index equal to our next location,
and our next location would need
the row index column index and action.
Lastly, we're looking at our rewards.
Our old Q values would be Q values
with the old row index, the old column index,
and again, referencing our action index
and the calculation of our temporal difference,
which is our reward plus the discount factor
times the Q values with the row index and column index
minus the old Q value from the previous Q value.
Almost done, where we need our new Q value,
which is our old Q value plus the learning rate
times temporal difference,
and our Q values with the old row index
and our old column index equal to the new Q value.
If you guys want to discuss this any further,
again, highly recommend the Q and A.
Please feel free to print.
Try to experiment with all the variables used in here.
If you wanna see the shapes, see how they can be used,
what it's referencing, I highly recommend it,
and I'm more than happy to discuss it further.
One last thing we can do so we have
a little bit of a notification.
Let's add a print statement so we know
when the training is complete for our thousand rows.
All right, if there's no syntax errors on my part,
we should be good to go.
The collab notebook is initialized.
I reran everything, so let me run this.
It should be very quick, since we're only using NumPy.
It's a very optimized model in a sense.
Let me click this, and we will then look
for the shortest path.
All right, we have our training complete.
Let's also print the shortest path for some starting option.
So we're starting at row three, column nine
with the shortest path option with our get shortest path.
We wanna look at row five and zero,
and starting at row nine and column number five,
let's print these out.
Awesome, we have the shortest path, but we're not done.
We're almost there.
We can see our postman automatically get the shortest path
from a legal, we can consider, location
from our city to the item packaging area.
But what would about the reverse to the opposite scenario?
Basically referring to can our postman deliver an item
from anywhere in the city to the packaging area?
But after the item, it would need to go then from that area,
from the packaging area, to another location in the city,
because it would have to pick up the next item.
So what can we do to solve this?
And it's actually pretty simple.
You could revise the order of the shortest path.
Try to think about that for a second.
And then, this would be the solution.
We can use the get shortest path.
So let's take row five and column two,
and then all you would need to do is use
the reverse option with Python.
We use path dot reverse,
and we could print out the path.
So we're looking at five and two,
and there we go, we have the shortest path.
And it's really helpful if you want
to grab the image in the cell
and bring it down to compare and look at those paths.
But you can actually see the paths
if you take the printed out, the statement,
look at the column and the order
and see how the salesman is traveling.
So awesome work; you are using Q-learning.
What we learned from this course has a little bit of a bonus
and fun scenario section to solve this,
essentially, traveling salesman kind of problem.
We're looking at the postman to deliver packages
and items in the city to find
those shortest options and shortest paths.
Highly recommend that you customize,
test out other options, change the hyper parameters.
And if you discover anything better,
if you discover any optimized or best parameters to use,
please feel free to share them in the Q and A.
Amazing, hope you guys really enjoyed this.
Please customize, experiment, and keep learning.
It's so much fun to work with Q-learning and enjoy AI.