-: Hello and welcome to this Python tutorial.
All right, so we have one last function
to implement in our replay memory class.
That's to sample function.
And, that's of course, to get some random samples
from our memory and therefore,
this function will return these random samples.
All right, so let's implement it.
We are going to call it, "Sample."
(keys tapping) Here we go.
And, this function takes two arguments as input.
The first one, as usual, self.
Our future objective, to replay memory class.
And, the second argument is...
Can you try to guess?
Well, we're taking some samples, a fixed size,
and therefore, we need to choose a size for our samples.
And more precisely, we call it a batch size.
So, that's the name we're gonna give to our second argument.
Batch Size.
And, there we go, we have our two arguments
and now we can implement the sample function.
So, now I just want to warn you,
this is going to get a little technical,
but I'll try my best to explain.
So, we're gonna start by creating the samples variable.
This is just the variable that will
contain the samples of the memory.
All right, so samples equal,
so now, how are we gonna get these samples?
Well, first of all we have to take our memory,
because we're getting these samples from our memory.
Then, we will probably need the batch size,
because the samples we're gonna get
contain batch size elements.
So, we need memory, we need batch size,
and then, we need some PyTorch, or Python tricks,
to get a good format of these samples.
So, what I'm gonna do, I'm gonna write the line of code,
and then, I'm going to explain it element by element.
So, let's do it. I'm starting by taking a zip() function.
I'm going to explain very soon, what it does.
And, inside this zip function,
I'm goin to add a star, I'm going to explain that, as well.
A star and "random.sample".
So, random, as you might have guessed,
is the random library that we import here.
So, that's the main reason why we had to
import this random library.
It's because we were taking some random samples,
so from this random library,
we're gonna use the sample function.
So, this is our variables and this is a function,
so I'm gonna add some parenthesis, and now,
as you can see, sample is a function,
and we have to input some arguments.
So, as you can see, the first argument is self,
and actually, speaking of self, this corresponds to
self.memory,
the memory of our future instance,
object of our replay memory class.
So, I'm gonna add here "self.memory",
and then the second argument is,
as you might have guessed,
the size of the batch we wanna take randomly
from our memory, and that we gave it a name.
That is batch size, so the second argument
is going to be batch size.
All right, so the line of code is typed,
and now I'm going to explain what it does.
So, first of all, with this random.sample function,
we're taking some random samples from the memory,
that have a fixed size of batch size.
So, that's understandable.
But, then what does this zip(*) function does?
Well, there is no mystery about it,
it is just like reshape function.
So for example, I am going to add a little comment here,
just to explain that I'm going to remove it.
So, let's say that, for example, we have a list
of the following elements, for example,
first, one, two, three,
and then the second element,
four, five, six.
So we have a list of two-tiples,
of three elements one, two, three, and four, five, six.
Well then, if I apply the zip() function with a star on it,
well, what will it become?
zip(*list)
That's going to be equal to a new list,
but of a different shape.
And, this different shape is going to be
one, four,
then two, three,
and then five, six.
All right, so that's just what it does,
it just reshapes your list. Okay?
So, now that you understand
what this zip(*list) does,
well, now let's explain why we have to do it.
So, as you understood,
we are gonna add the events to the memory,
and the events have to form first, the state,
then the action,
and then, the reward.
But, for our algorithm, we don't want this format,
we actually want our samples to have the following format,
formats composed of three samples,
one sample for the state,
one sample for the actions,
and one sample for the reward.
So, for example, let's say that this, "One, two, three"
is state one, action one, and then reward one.
And then, state two, action two, and reward two.
Well, what we want is, one batch for each,
one batch for state one and state two,
one other batch for action one and action two,
and a third batch for reward one and reward two.
That's just a format that is going to be expected next,
because then we will wrap these batches
into a PyTorch variable.
And a PyTorch variable, remember, is a variable
that contains both a tensor and a gradient,
and that in order to be able to differentiate
with respect to a tensor.
To be able to differentiate with respect to a tensor,
we need the structure of a variable
containing a tensor and a gradient.
Again, that's how PyTorch works.
So, to summarize, we're creating one batch
for each of the states, actions and rewards.
And then, we're gonna put each of these batches
separately, into some PyTorch variables,
which each one will get a gradient.
So that eventually,
we'll be able to differentiate each of them.
All right, so that's the purpose of the zip() function.
So, let me just remove this comment,
and now, the only thing that we have to do left,
is to return the samples.
So, as I just explained,
we cannot return the samples directly,
for the simple reason,
that we want to put the samples into a PyTorch variable.
So, to do this for each of the samples,
we're gonna use the map() function, and this map() function,
will do the mapping from the samples to Torch variables,
that will contain a tensor and a gradient.
So, as you can see, this map() function
takes several arguments.
The first argument is a function, and this function
is going to be the function that will convert
the samples into some Torch variables.
And, the second argument is what we want to apply
this function on to, so that will be
the argument of this function.
And therefore, what is it going to be?
That's of course going to be the samples.
So, the second argument here is going to be the samples.
But then, let's define the function,
on which we want to apply each of the samples.
So, to define a function here,
we need to first give a name to the function,
which we'll call, "Lambda".
That's just a name I'm giving,
lambda, then "x",
which is going to be the variable of this function,
so, that is just a name I am giving for the variable.
And then, Colin.
And here, we give the expression of the function.
That is, what we want this lambda function to return.
And so, what it is going to be?
Well, it's supposed to be something that will
convert our samples into a Torch variable.
And to do this,
we already mentioned it in some previous tutorials,
well, we have the variable function for that.
The variable function will make that conversion
from a Torch tensor,
to a variable that will contain
this tensor and the gradient.
So, the first thing I'm gonna add here, is "variable".
Variable, inside of which, I'm going to convert "x",
because "x" is going to be the samples,
once lambda will be applied onto the samples.
But then, that's not all, there is one last technical thing
that we need to implement.
That is the fact, that for each batch,
which is contained in a sample.
For example, the batch of the actions,
A1, A2, A3, and the other actions,
we have to concatenate it,
with respect to the first dimension,
which corresponds to the state.
And, why do we have to make this concatenation?
It's just for everything to be well aligned.
That is, that in each row, the state, the action,
and the reward corresponds to the same time_t,
so that eventually, we get a list of batches,
all well aligned, and each batch is a PyTorch variable.
So, how can we make this concatenation?
Well, we need to use the cat function,
from the Torch library.
So, we're gonna add here,
"torch", to which we add ".cat" applied to "x".
But then, in this cat function,
we need to specify the dimension,
with respect to which we want to make that concatenation.
And, as I just mentioned, this is the first dimension
that has exact zero.
And, here we go, we have our function ready.
This lambda function will take the samples,
concatenate them with respect to the first dimension,
and then eventually, we convert these tensors
into some Torch variables,
that contains both a tensor and a gradient.
So that later, when we apply the casting grade,
in this sense, we will be able to differentiate,
to update the weights.
All right, so this function is ready, and then here,
at the second argument of the map() function,
we need to specify onto what
we want to apply this lambda function.
And, that is on all our samples.
There we go, we will apply this lambda function
on all the samples, so that eventually,
we obtain a list of batches,
where each batch is a PyTorch variable.
All right, so that was quite technical,
but now at least, everything will work well.
We won't use this technique afterwards,
we only use it here,
so if you don't want to have a deep understanding
of the technical details, here.
Well, that's fine, you can just
copy these three lines of code to sample your memory,
if you want to make an artificial intelligence with PyTorch.
It's as you want, but now, the good news is
that we are done with this replay memory class.
Experience replay is now implemented,
and we can move on to the next and final class,
which will be the whole, deep curating model.
So, in this deep curating model, we will have,
of course, our network.
We will add experience replay,
and then all the rest of the deep curating algorithm,
so it's going to be a much bigger class.
We're gonna make about 10 functions,
but that's only because we are doing this step-by-step,
so that you can understand better, what's going on.
So, I can't wait to implement our deep curating model,
and until then, enjoy AI.