In this lecture we are going to discuss the layout and design of our reinforcement learning trading
but first at a very high level we are going to have two modes of operation.
Train and test.
As usual we want all of our training data to be in the past and all of our test data to be stock prices
that came after the training data.
So we are going to train our agent to maximize its reward over an episode using only the training data.
Then we are going to use this trained agent on the test data to see what the value of our portfolio
is.
By the end of the test period
to start let's think about how this would work if we had access to all the objects we needed.
Even without the agent this will help us organize the majority of our code.
The main part of the code will look something like this.
First create an instance of the environment next create an instance of the agent.
Don't worry about what this does yet then in a loop we're going to have a function called play one episode
which accepts the environment and the agent and returns the value of the portfolio at the end of the
episode.
When our loop is done we're going to save the portfolio values for later so that we can plot them and
analyze them.
So this is pretty simple but now we have to figure out what should go in the function play one episode.
So here's what the play 1 episode function might look like.
As always we start by resetting the environment to get back to the initial stage.
Next we initialize our done flag to false and enter a loop that only quits when done becomes true inside
the loop.
We choose an action.
Now at this point you know that this action is coming from our agent but will defer how the agent works
until later next.
We're going to call that view that step function to perform the action and get back the next day reward.
And so on next we're going to check if our script is in train mode if it is then we have to train our
agent.
We want to update our replay buffer by adding the most recent SARS prime and done tuple to the replay
buffer then we'll call Agent replay to grab a sample from our replay buffer and run one step of gradient
descent finally we'll set the current state to be the next day for the next iteration of this loop when
we're done we'll return the value of our portfolio
one additional detail to keep in mind is that our data is not yet normalized.
You can imagine that our state which is composed of three parts can have vastly different ranges.
The first part consists of the number of shares we own.
The second part consists of share prices in the third part consists of how much cash we have sitting
on invested.
So we'll want to normalize this data.
We can do this very simply whenever we get a new state.
We'll have a scalar object from psychic learn which will take our state and standardize it to have zero
mean and unit variance.
So not a huge addition to our previous code
next.
Let's imagine what our environment objects will actually look like.
First it's going to accept a time series of stock prices as input into its constructor we'll also have
a pointer to tell us what day it is.
So we know the current stock prices while also want to know how much cash we initially start with our
initial investment from this we can do everything we need to do our reset function will bring our pointer
back to the beginning of the time series and recalculate our stay which of course should be all cash
and no investment our step function will take taken an action and then buy and sell the stocks specified
by the action then it'll set our pointer to the next day's stock prices.
We'll also calculate the next day and the portfolio value.
From this we can calculate the reward.
The done flag will simply be set to true.
If we reach the end of our time series so that's basically it for the environment we have the Constructor
a reset function and a step function
Finally let's consider our agent object which is complicated but no more complicated than the environment
the essential parts of the agent are.
The replay buffer which is what we use to store our transitions in the environment and a neuron that
work or some other kind of model which is what we will use to approximate our Q values.
So what are the essential functions of the agent as we've seen.
We'll need a one function called update replay memory which takes in a state action reward next day
and done flag and stores this in our replay buffer object.
Next we'll need a get.
Action function which accepts as input a state and decides what action to perform in the environment.
And because this is Q learning it's going to use the Q learning rule or some variant of it like Epsilon
greedy finally we'll need a replay function which does the following.
First it grabs a random sample from the replay buffer next.
It uses this to calculate a supervised learning dataset which consists of input and target pairs which
is what we need to train our model once we have our data set.
We can call model that train to run one iteration of gradient descent.
All right so that's it for this lecture.
There are certainly more details we haven't yet discussed since this was designed to just give you an
overview at this point.
You understand that our script will have a train mode and a test mode.
In both cases there will be a main loop where we call play one episode again and again playing when
episode involves basically just going back and forth between the agent and the environment.
The environment produces states and rewards the agent takes in states and returns actions to perform
in the environment during train mode.
The agent will store the state's actions and rewards and perform.
Q learning updates in order to train the Q function approximate Herr.