In this lecture we are going to describe the agent the agent is our artificial intelligence.
It's responsible for taking past experiences learning from them and taking actions such that they will
maximize future rewards.
First we have our constructor it takes into two arguments the size of a state vector and the number
of actions.
These correspond to the number of inputs and outputs of our neuron that work respectively.
Next we initialize an instance of the replay buffer object with the memory size of 500 marks we have
a few hyper parameters such as gamma the discount rate the initial value of epsilon the final value
of epsilon and the factor to change Epsilon by on each round.
Lastly we create an instance of our model by calling the MLP constructor.
We also create the associated loss and optimizer and make these attributes of the agent.
Next we have the update.
Replay memory function.
This takes in a state action reward.
Next state and done flag and stores it in the replay buffer.
Next we have the ACT function.
This takes an estate and uses Epsilon greedy to choose an action based on that state.
First we generate a random number between 0 and 1 and check if it's less than epsilon.
If it is we perform a random action by calling n picked out random nut choice.
Otherwise we perform a greedy action by grabbing all the queue values for the input state then the action
to perform is the action which leads to the maximum Q value.
So we take the ARG Max over the model predictions remember that the output of a PI to each model is
backsides by number of outputs.
So we have to index the return value by zero before taking the ARG Max.
The batch size is just one.
So we're looking at the 0 with index.
Next we have the replay function which is the most important function for this class.
This is the function that does the learning.
It accepts one argument the backside which tells us how many samples to grab from the replay memory.
So to start we first check if the size of the replay buffer is greater than the batch size.
If it's not we can't grab a full batch.
So we just return.
Otherwise we continue.
Next we call self memory that sample back with the batch size argument.
Remember this returns a dictionary so we grab the state's actions rewards and so on using the corresponding
keys next we calculate the targets for each state for reference.
Here is the equation again.
Mathematically it's a little different here because we are now working with a batch of data rather than
just one sample.
Also we have to remember that by definition the value of a terminal state is zero.
Therefore if the next state is a terminal state then why should just be the reward.
Ah to that end we can multiply by one minus done since day one equals one if it's a terminal state then
you would get one minus one which is zero and anything times zero zero.
So effectively the target would just be are the reward
now.
Interestingly this may or may not be correct for the scenario.
Realistically the stock market never ends and so you don't really have a proper terminal state.
This is only where our data ends.
So one might theorize that it may be better not to do this saying that this is the end of the episode
is sort of pedantic.
It's not really the end of the episode it's just the end of our data.
The stock market will effectively go on forever.
Of course you can just test it out and see if it has any effect
now at this point even though we've calculated the target.
We are technically not done currently.
The targets are just a 1 b array of length back size but the model predictions are a 2D array of batch
size by number of actions so they don't match.
The problem is the targets and the predictions don't have the same shape as always.
I'll remind you how important it is to think about shapes.
Now one option to solve this problem would be to just use a custom lost function but if you would like
to use the default means squared error then the targets and the predictions have to be the same shape.
In other words for each sample we must have a target for each action.
Even if that action was not the one taken by the agent in order to do this we're going to use our model
to make a prediction for each state.
Any action so we'll call that targets full.
So this will be batch size by number of actions.
Now the key is we only want to change this array for actions in which we really have targets.
So the targets we calculated above to do this.
We can use double indexing where the first index which is empty are a range of that size.
And this represents which row to choose the second index which is actions tells us which column to choose.
And we can set these to the targets we calculated previously.
The trick is for all other actions the target will be equal to the prediction because the target actually
is the prediction therefore the error of those values will be 0 and they will not have any influence
on the gradient descent step.
Next we run one step of gradient descent using the function to model that train on batch passing in
the inputs and targets we just calculated finally we update Epsilon to reduce the amount of exploration
over time.
If Epsilon is still bigger than epsilon men then we decrease Epsilon by a factor of epsilon decay
after this we have two simple functions to save and load the model weights.
These will be useful since the script takes some time to train.
So what we can do is train the scripture in one run save the weights and then test it later with different
configurations.