Instructor: Hello everyone and welcome back.
In this video we'll talk about how to use ChatGPT
to solve "Breakout,"
using ChatGPT to implement A3C.
So I already prompted it with a lot of crazy prompts,
we got amazing results, we'll see them in a second.
So this video will be basically walking view
through my thought process
on how I actually got the results,
and how you can actually use ChatGPT
to create A3C algorithm as well.
Okay, so I started similarly to what we had for "DOOM,"
I stated, okay, you are basically
a machine learning expert
with years of experience in robotics.
I hire you to do A3C algorithm for "Breakout."
The goal is to have
a fully functional SOTA, A3C algorithm
that can play the "Breakout" game.
I have implemented classes,
the testing code environment,
and custom versions of Adam optimizer
that is used for as a shared-weights optimizer
for the A3C model.
So this is giving it more context, what I did,
I will provide these three things to you.
So as I reference before generating
the rest of the code,
your goal is to implement
the full solution using Python, Torch,
similarly to what we had,
and optimize the code to be executed in Google Colab,
same what we had before.
Additionally, comment every single line
of the code so the students can understand
what you did in the code.
Ask clarifying questions if needed, got it.
So we have some overlapping
what we did for conversational Deep Q for "DOOM,"
but I added a lot of context
that I already have things implemented,
and I'd like them to be used.
So that's what I start and it's confirmed.
Yes, I understand and I'll ask any questions,
please provide the rest of the code.
Then I stated, this is Adam optimizer,
pasted the code for Adam optimizer,
for the resources we have,
and I gave this instruction,
I said, wait for the rest of the code before implementing,
I just, before I got any good results,
I actually just pasted
and it immediately jumped to implementing A3C,
without waiting for the rest.
So I added this as a kind
of systemic code, systemic prompt,
and it said, okay,
please provide the environment
and test the code.
So it's actually referring
to what we gave in the first prompt.
I said, cool, here is the implementation
of Gym environment.
I pasted that main code,
I added context here,
if you take a look,
and this is the main file running everything.
And then I said, okay, cool,
wait for the testing code.
And it said, yeah, I,
however, I still need to see the testing code
before I implement A3C,
I said, cool, here is a testing code.
And you see this is how if you have
a bigger, bigger project,
you can use that
to kinda sway it to use
that piece of a code or as a reference,
in the style so you can actually generate something
in your style of code.
Cool, and when I done that,
it said, great, go for it.
And it said basically I'm going to use this main code,
and testing, and implement this.
And then it said, created model.py,
which actually we did in the course as well.
And he said, here is the ActorCritic model,
and I compare to ours.
This is the newer version of python,
so much of the weights initialization
and stuff like that can be done in a more elegant way.
So it did it and it actually uses pretty well,
then generated train code,
and basically ActorCritic.
And if you take a look,
it's a pretty simplistic,
so it definitely needs to be improved.
So what you can do,
you can say, okay, I said continue,
and it continues writing it down.
It created this
as a kind of a loss,
but there is, yeah,
basically everything
that we needed, it optimized everything,
and you can see that optimizer.stop at the end.
So it basically used everything
that we defined so far,
as well as the proper loss,
and yeah, optimizer, everything.
So that's it.
In a couple of simple steps,
we managed to implement the whole code
for one of the state-of-the-art models
that happened to be like one of the best
in 2017, 2018.
To this day, it actually hold a lot
of capacity to solve complex environments
such as "Breakout."
So you saw, now that we have ChatGPT for example,
you can guide it to implement certain
certain parts of the code.
And if you don't know what certain parts
of the code actually does,
you can go and ask it,
for example, cool,
can you explain training part
and where is the custom
of Adam?
So I'm prompting it to actually explain itself,
so that's another way
to basically prompt it out
to give you more reasoning behind its generation.
And you can see
that it's actually going to explain step by step.
So yeah, we can wait for this,
you can do the same prompts
and you will get really similar results to mine,
but generally now what it's going to do
it's going to take the train function
that it defined right at the top right here,
and it'll basically go through every single step
and implement it.
So yeah, that would be it.
I definitely encourage you
to try out this code in Google Colab,
It can be done,
It'll be a bit too tricky to make it work,
because of the visualization
and limitations of the Goggle Colab environment.
However, you can definitely go
and paste the errors here, and you can,
you can go and get some pretty decent results
for that to debug the process.
And yeah, with that I'll let you be,
and now you have the way to use ChatGPT
on how to get value for A3C.
And yeah, hope you enjoyed
the course so far, bye.