-: Hello, and welcome back
to the course on artificial intelligence.
I hope you're excited about today's tutorial
because we are taking our very first step
into the world of AI.
And today, we're talking about reinforcement learning.
It's a very important tutorial
because it will underpin everything else
that's going to be happening in this course.
So, let's get started.
Here we've got a little maze,
and this maze is our representation of an environment.
And that's what we're going to be dealing
with in this course.
We're going to be dealing with certain environments
in which our artificial intelligence
is going to be performing,
is going to be taking actions,
is going to be looking to beat these environments
going to be looking to win in these environments.
And here, we've got an agent.
The agent is our artificial intelligence.
That's the person,
or that's the mind that's going to be navigating
these environments
and learning from the feedback
that the environments are going to be giving it
in order to perform certain actions.
And so the way it works
is the agent performs certain actions in this environment.
And as a result, the state in which it is in will change.
So, it might be further or closer,
or more to the left, more to the right.
It might have a certain other parameters
that describe its state.
And those parameters are going to change.
So, the state is going to change
because of the action it takes,
and it will also get rewards based on the actions.
So, every time it takes an action,
the state will change and it'll get a reward.
Now, bear in mind sometimes it might happen
that it won't change the state,
the action won't change the state,
or there won't be a reward
for taking that action in that certain state it was in.
But nevertheless, the agent's going to keep doing that.
It's going to be taking actions,
changing the state, getting rewards, changing action,
taking actions, changing the state and getting rewards.
And by doing that process,
it's going to be learning about the environment,
it's going to be exploring the environment,
understanding what actions lead to good rewards,
and favorable states,
and what actions lead to bad rewards and unfavorable states.
And this is a very simplistic representation
of a very global problem.
So, if you think about it,
environments actually don't have to be just mazes.
It's not just about getting out of a maze
or finding a treasure in a maze.
An environment can be pretty much anything in life.
So, imagine you waking up
in the morning and cooking an omelet.
So, in order to make that omelet,
you need to go through certain steps
You need to get the salt, get the eggs,
get the frying pans,
switch the fire on, and so on.
And it does sound like a routine mundane thing,
but it's become routine
because you've done it so many times.
But in reality, it's an environment
where you're performing certain actions.
You're putting the fire on,
you're putting the frying pan on,
the fire, you're putting all the eggs into the frying pan,
and you're putting some salt on the eggs,
and you're turning them over and so on.
So, as you can see, there are certain actions
which you're taking in certain states.
And those actions lead to certain other states
and sometimes rewards.
So, for instance, when you put the fire on
and you wait, wait, wait, wait, wait,
you're taking the action of wait, wait, wait, wait too long.
And then you put the eggs into the frying pan,
the rewards are gonna be very negative.
It's all gonna burn.
On the other hand, if you do all the correct actions
in the correct times...
So, it's also very important to understand
that actions should be taken at the correct points in time.
So, for instance, putting the salt in the frying pan
before you put the eggs in might not be the best idea.
You might want to take that action of putting the salt
into the frying pan after the eggs are in there
so in the different state.
So, it's important to remember that.
And at the same time,
if you take all the correct actions
in the correct order, in the correct states,
your final reward could be that you get an omelet
which you can eat.
And so that's a very basic activity in your life.
But if you think about it,
it is actually an environment
and you are the agent going through this environment
and performing task.
You don't really need to learn anything
because you already know it pretty well.
But at the same time, you could learn.
Maybe you could learn how to make a better omelet.
Or especially if it's your first omelet that you're making,
you probably gonna screw it up.
But you will learn from that
because you will understand what actions lead
towards states and rewards.
And anything else in life, for instance,
even trading on the stock market,
and you know,
buying and selling, and getting certain feedback
from the market in the sense of return,
positive or negative returns, that's also an environment
and that's you participating
in that environment as an agent.
Driving a car is also an environment
where you can turn the steering wheel,
you can accelerate, you can break, and so on,
and you're getting feedback from the environment.
And you know,
one of those feedbacks is the policeman
giving you a speeding fine
if you're going above the acceptable
or allowed speed limit on that highway.
And therefore from there, you learn that,
"Okay, that's not something that should be done
because it leads to a negative reward."
So, rewards don't have to be just
at the very end of the process.
They can be throughout the journey, throughout the process.
So, those are a couple of examples.
And in terms of AI,
the simplest way to think of reinforcement learning
is like training a dog.
When you train a dog, you give it certain commands,
and if it obeys those commands, then you give it a treat.
You give it like a biscuit or something.
If it doesn't obey those commands,
you tell it that it's a bad dog
or you just don't give it a treat.
And through that process,
it learns what certain commands or what it needs to do,
what action it needs to take in certain states.
And the states are the commands that you're giving it.
And based on that, it'll get some certain rewards.
Of course, in the world of AI, it's not that complex.
You don't have to give the AI treats,
you don't have to have like a bag of biscuits
with you every time.
You just give it a plus one or a minus one.
So, it's a huge advantage that in the world of AI,
we've created these AIs ourselves.
So, the rewards that we're giving them,
if you think about, this is really cool,
the rewards you're giving them, they don't actually exist.
They're just a plus or a minus one,
or a one or a zero, something like that.
So, it's all non-existent, it's all imaginary stuff.
But at the same time, it leads to great resources.
We can create these amazing things,
by this amazing artificial intelligence
by just providing rewards which don't really exist,
the plus and minus one.
It doesn't cost us anything
but at the same time, that leads to results.
So, very similar to real world,
and you know, that example of dogs,
but here, the rewards are digital and just numbers.
And with that in mind,
we can talk a little bit about Robodog.
I love this example.
So, this is just a random picture.
It's not necessarily that exact Robodog,
you know, that is trained through enforcement learning.
Some Robodogs, especially the older ones
you'd have an algorithm in there.
And this is actually a good example
of the difference between pre-programmed agents,
and reinforcement learning agents.
So, you could have a Robodog
which is pre-programmed to how to walk.
It will say...
So, in the algorithm behind the dog
and the software will say,
"Okay, so in order to walk,
you need to move your left leg forward,
front leg forward, then your back right leg forward,
then your front right leg forward,
then your back left leg forward, and repeat that action."
And you know, that's a definition of walking.
It's a function inside this dog.
And then it might have, you know, how to sit,
how to stand and things like that.
Whereas in a Robodog that is trained
through reinforcement learning,
what happens is you don't pre-program it.
This is the key concept to everything here,
that you don't have any algorithm
inside that is hard-coded into the dog.
Instead, you have what we'll be discussing in the future.
You have this reinforcement learning algorithm
which is told that,
"Okay, so the goal is to get from where you are now,
not knowing anything to the end of the room," for example.
"And here are the certain actions you can take.
You can move your right foot, you can move your left foot,
you can move your right back foot, or left back foot.
So, here are all the degrees of freedom that you can do.
You can move them like this, you can move them like that."
So, like a list of actions you can take.
And your rewards are every time you take a step forward
you get a plus one.
Every time you fall over, you get a minus one.
And that's all there is to it.
And then they just leave the dog
and let it figure it out on its own.
So, the dog tries to stand up, it falls,
and it realizes that,
"Okay, I shouldn't do that action that led to me falling,
because every time I fall, I get a minus one
which is not good for me."
Then, so it does the other action that helped it stand up.
And then it figures out, it just experiments,
experiments, experiments,
tries things randomly, and then figures out
that you can make a step forward
by moving its right front foot.
And then, and it gets a plus one and it realizes,
"Oh, I should do more of that."
Okay, cool.
So, it now learns that it should do more of this
and less of that.
And through this learning process,
it very quickly understands how it can walk.
And those dogs that figure it out on their own
can actually sometimes walk better
than dogs that are pre-programmed.
Because when we pre-program things,
we look at the real live dogs,
and or you know,
we use our own imagination how to do it.
Whereas a reinforcement learning dog
can optimize things on its own.
And because it's in AI,
sometimes it can get even better results.
And that's how they can train
these same Robodogs to play soccer.
You can't train a normal dog to play soccer
because you know, simply the whole approach is different,
and it's not something that you know,
probably a normal dog has been trained to do,
or has ever done in its process of evolution.
Whereas a reinforcement learning Robodogs,
can very easily understand how to play soccer
as long as you tell them what the rewards are,
what the goals are,
what the possible actions they can take are.
So, that is how reinforcement learning works in general.
That's a quick overview of reinforcement learning.
I hope that got you very excited
about what's going to come next,
because it's a completely different world
compared to pre-program solutions,
or hard program, hard coded solutions
where you have the if-else conditions.
This is very different
and we're gonna be talking more about that.
In the meantime, we've got some additional reading for you.
So, if you'd like to have some supporting materials,
here's a great article which you can look into.
It's called "Simple Reinforcement Learning with Tensorflow."
It's got 10 parts.
The link is here and you'll find the clickable link
in the course resources.
It's by Arthur Gillian.
It's a 2016 article.
And you can follow along this course
and also get additional information from that article.
But bear in mind that that article is with Tensorflow,
whereas in this course we are using PyTorch.
So, different implementations
but at the same time,
you might pick up a few things here and there
that might supplement your learning
that we're gonna be doing in this course.
So, great article to follow
even if you're not considering following it for sure.
Still just in case,
check out that first part and see if you like it.
See if you would like to read it a bit more.
And then we've got, specific to this tutorial
about reinforcement learning,
there's a paper by Richard Sutton,
which is called "Reinforcement Learning One Introduction."
It's a 1998 paper, so quite old,
but at the same time,
you can learn a bit about reinforcement learning,
some of the examples like that omelet example
and other examples
of where reinforcement and learning can be applied.
And just a general overview of reinforcement learning
if you are looking for some additional reading.
And on that note, we're going to wrap up this tutorial.
Can't wait to see you next time.
And until then, enjoy AI.