-: Hi guys. Welcome to the final tutorial for Breakout.
-: Hey guys, super excited to have you here.
-: And super happy to be all together again after Doom.
-: Yeah.
-: So this time we're gonna watch the result for Breakout
which I remind was much more challenging than doom.
So let's hope we get some good results.
-: Okay.
-: Yeah. -: So what are we doing?
-: So here is the main code.
-: Yeah.
-: So, you know, we have several codes.
The, all the codes are here,
-: Mhmm
and here is the main code.
-: Yeah
-: The code that will execute all the other codes.
So build the new network and train the, the agents.
-: Yeah.
-: And the critic., okay?
So basically what we're gonna do is just execute
just to see how it's going,
but we will not wait for the results
because actually it takes several hours.
-: Yeah.
-: So we, we prepared the results in this test folder.
We already have the video
and we will show you right after that.
-: Okay.
-: Okay?
So to execute, basically we just select the whole code here
and here we go.
Yeah. -: All right.
-: And as you can see, the videos are populated.
-: Okay.
-: All right.
So right now we already have two videos
and then the agents are training
but it's gonna take a long time.
So now what we're gonna do is watch the results.
-: Okay.
-: Which I haven't seen yet.
-: You haven't seen it yet, yeah.
-: All right.
-: Right.
So, in the template folder that you will find
on the SDS website, you will find two folders.
-: Mhm.
-: Code no comments,
and code with comments.
So that's, even if you missed any detail
you can still have a look at this code.
(mouse clicking)
You have all the comments well explained.
So, each line is well commented so that you can understand.
And then in this folder, we'll find this test folder
and that's the folder that will be populated
with the videos.
-: Okay.
-: So that's where the videos are.
So this-
these are the videos of our previous training.
You know, we trained the model for a couple of hours
and we obtained these videos.
So we're gonna watch them one by one
starting with the first one.
-: Mhm.
-: And we're gonna see if,
you know, -: Yeah
-: we see some artificial intelligence playing breakout
and if we manage to solve breakout.
Okay? -: Okay.
Let's have a look.
-: So let's have a look at the first video.
So this first video is, you know...
Actually what we're gonna do.
Yeah.
What we're gonna do is stop the training
because it's slowing down my computer right now, so, okay.
-: Mhm.
-: So interrupted,
but all good.
-: Yeah.
-: So let's watch the first video.
It's gonna be really bad because it's really the first one.
The, the pedal might not even move.
There we go.
Yeah, it doesn't move.
-: Why doesn't it move at all?
-: Because you know, it's the very beginning.
The AI doesn't know what to do.
-: Yeah.
-: And it's still pre-processing the images, so yeah.
Nothing. -: Gotcha.
-: And it might even be the case for the second video.
If we look into the second one
the paddle might not,
might still not move.
-: Okay.
-: So how, what do you think of this
artificial intelligence so far -: A3C sucks(chuckles).
(laughing)
-: So that was worth spending so much energy
on the A3C, right?
-: Yeah (chuckles).
-: Okay.
So next video, we might have something.
-: All right, let's have a look.
-: Maybe a little something,
a little bit of artificial intelligence.
-: Okay, you're giving away too much.
(chuckling)
-: Not even.
-: Not even, no.
-: Not even.
-: Useless.
-: Are we gonna break breakout or...
-: No.
-: No?
-: All right. Come on. Let's do something,
-: Okay
-: Let's, let's hope that we get something for the next one.
Ah, yeah, I think we get something
because now that's 11 seconds.
-: Okay.
-: There we go.
-: Up.
-: Better, okay.
So now the AI has eyes, it can see.
-: Yeah.
-: It can sense what it's doing.
-: Yeah.
-: But it's not very quick yet.
-: Yeah.
-: It's not a smart AI.
-: Okay gotcha.
-: But better.
-: Yep.
-: Okay, so four.
-: Four.
-: Okay.
So next video now.
-: Mhm.
-: Ooh, it looks much better now.
-: Yeah.
More time.
-: Yeah.
Are you ready?
-: Yep. Let's have a look.
-: Yeah.
-: Okay.
Got it.
I got another one.
Ah, no, you lost it.
-: Yeah.
Still not perfect, but definitely it's like a human playing.
-: Yeah.
-: Not very good at breakout, but still it's like a-
-: Like a
-: human playing.
-: like a five year old, two year old human playing.
-: Yeah yeah.
-: Okay.
Wow, It's actually getting them.
-: Yeah, a good two year old now.
-: Yeah.
-: But it's like, it's doing, you know
trying to catch the ball with some weird tactic.
-: Yeah.
-: It's not like a perfect
prediction of where the ball is going to be.
-: Gotcha. Okay.
-: Okay.
-: Let's hope it gets better.
-: So much better at 27.
-: Uh-huh.
-: Okay.
Next one,
37 again.
-: Oh, okay.
-: Okay.
-: Yeah, that was 27, this is 37.
-: Okay. No, I mean the time.
-: Okay.
Oh okay.
-: Yeah.
Now I would say it's a five year old.
(chuckling)
-: Okay.
-: Hmm. Pretty good, huh?
-: Yeah.
-: And now the goal, the goal is going faster.
-: Mhm.
-: So it's of course much more difficult.
But still, it's doing great.
So let's see if we can beat 27 this time.
23, 24,
28. Okay, here we go.
See it's getting better and better.
-: Yeah. Yeah.
-: That's a good sign.
-: Mhm.
-: Okay.
-: Cool.
-: Lasted three lives.
Next one.
(mouse clicking)
Whoa.
One minute, 10 seconds.
-: Oh okay.
-: Let's see.
-: Straight there.
That's really cool.
-: Yeah, that's definitely a pro now,
isn't it?
-: Yeah. And you can tell it's not a human playing
because it's like it's moving sporadically like super fast.
-: Yes.
-: And it's doing a lot of,
I would say even unnecessary movements,
but it's still getting there.
-: Yeah.
Let's see how it does when the ball is going really fast.
-: Where does the ball speed depend on?
-: Um.
I don't know. I think it's a....
-: Does it depend on the color of the block it destroys?
-: Maybe.
-: Or is it just like random?
-: Mm, it looks like random but
we should check out the breakout rules.
-: Yeah.
-: But it's, it's really doing great now.
-: Yeah.
-: 34. 34 score.
-: Nice.
-: Wow. It's really, really doing great.
So yeah we definitely have
some A3C playing and breakout now.
-: Nice.
With an LSTM, yeah?
-: Yeah, this was an LSTM.
So it's like we have a brain with a memory.
-: Yeah.
-: Not only with eyes.
-: Mhm.
-: It has good eyes, you know?
It's a quick, pretty quick
-: Yeah.
-: to catch the ball.
But also it's got some memory, but
I don't know.
Would you use your memory to play breakout yourself?
-: Of course.
-: Yeah.
-: Okay.
So, well done.
Well done to the A3C, the AI. -: Okay. What's the,
what's the last one?
-: The last one?
Oh.
55 seconds.
So a little less.
-: Yeah.
-: You remember what the score was?
-: No.
-: On the previous one?
Let's check it out to see if we get some improvement.
We might not get some improvement every video.
-: Yeah. Yeah. Exploration.
-: Okay, so that was 78.
-: Okay.
-: Yeah, that's due to exploration as you say.
Okay, there we go.
-: Hmm hmm hmm
Okay.
Think we should have like some music playing while was-
-: Yes (chuckling).
Like hypnotizing.
You can do it.
It's like on steroids.
Clicking left to right all the time.
-: But it's doing good.
-: Yeah.
-: But we might not solve the whole game of breakout but-
-: Yeah
-: we just need to have it trained more.
-: Yeah, that was already really good.
-: It's really good, yeah.
Oh boy -: That speed.
-: Oh yeah, that was a little bit too fast.
-: Sixties.
So it beat the score again.
-: That was 78, wasn't it? -: I know.
-: Yeah.
Okay, so we did it, didn't we?
-: Good stuff.
-: Definitely some AI playing breakout.
-: Yeah. Nice.
Very nice, very nice.
Awesome.
-: So I hope you liked it.
And mostly, I hope you liked this powerful implementation
of the A3C
and um-
-: Yeah, you can see what how, how it can do really well
with the LSTM.
Yeah?
-: That's right.
Like at- at the end of the A3C paper,
they have like the comparison of different algorithms
and this one is the best one by far.
-: Not only they have the best,
the comparison of these models
but also I experienced several of them.
And so I experienced one without and LSTM and one with
and definitely the LSTM improved the algorithm.
-: How much better is it within those?
-: Much better.
Like I, I barely some artificial intelligence
before playing breakout.
It last like 15 seconds.
-: Okay.
-: Or 20 seconds.
But definitely not more than one minute as we saw here.
-: Yeah yeah.
That's good, that's good.
Okay, cool.
-: Cool.
Thanks a lot guys.
-: Thanks a lot.
That was the end of the third module
and therefore the end of the course.
So, congratulations for having competed the three modules
and mostly the A3C, which was a pretty big (chuckles).
Pretty big.
-: Yeah yeah.
And for sure explore the different modification that,
modifications that exist for the A3C.
-: Yes.
-: And you know, maybe at the time you're watching this
there's some really cool new advancements
in A3C and you can have some-
-: Probably probably,
or even a new model even better than the A3C.
-: Yeah
For sure.
-: Let's see what 2018 says.
-: Okay.
All right. -: All right
Thanks all you guys.
-: Thanks.
Bye.
(piano music)