In this lecture, we are going to finish looking at the rest of this script and take a look at the results.
First, we have the play one episode function, which is pretty much exactly the same as what we discussed
in the theory outside the loop.
We reset the environment, grab the initial state and transform it.
Next, we initialize Dunn to false.
Next, we enter a loop which exits only when Dunn becomes true.
Next, we use the agent to determine the next action.
Then we perform the action in the environment.
We get back the next day.
Reward Dunn Flag and Info Dictionary.
Next, we scale the next state.
Next, we check if we are in train mode, if we are, then we add the latest transition to our replay
buffer and we call the replay function to run one step of gradient descent.
Next, we set the state variable to next day for the next iteration of the Loop.
Lastly, when we exit the loop, we return the current value of the portfolio.
Next, we have the main section first, there are a few configuration variables, so Models folder tells
us where we will save our models.
Rewards folder will tell us where we will store our rewards from both the training and testing phases.
The number of episodes to run is 2000.
The batch size for sampling from the replay memory is thirty two and our initial investment is twenty
thousand.
Next, we create an argument, the object, so that we can run the script with command line arguments.
We'll have one argument, the mode where we can pass in a train or test.
Next, we create the model directory and rewards directory in the case where they do not yet exist.
Next, we call the getData function to get our Time series.
We call the shape attribute to get the number of time steps and the number of stocks.
Next, we split the data into train and test, the first half is train and the second half is test.
Next, we create an instance of our environment objects with the training data.
After that, we get the dimensionality of the state and the dimensionality of the action space.
We pass these into the Asian constructor to get the agent.
We also called get Scaler to get the scalar objects.
Next, we initialize an empty list, which will store the portfolio values for each episode we play.
Next, we have an important if statement, this is going to overwrite some of the things we just created
in the case where we are in test mode.
So if we're in test mode, we want to use the scalar that we had during training so that the neural
network corresponds with the same scalar and we don't accidentally use a different scalar.
Next, we recreate the environment with the test data rather than the training data.
Also, we have to make sure to set Epsilon to a small value.
Otherwise its default value is one which is just pure exploration.
Note that if you set it to zero, there is no point in running multiple episodes because both the data
and our agent will be deterministic.
So you'll get the same result each time.
Finally, we have to make sure we load up the weights that we saved during trainings, we call agent
load and pass in those weights.
Next, we enter a loop to play our episodes, some episodes, times Inside the Loop.
We grab the current time since we would like to know the duration of each loop iteration.
Next, we play one episode and receive the portfolio value at the end of the episode.
Then we get the current time and subtract the start time to get the duration of the episode.
Next, we print out all this information, the episode number, the portfolio value and the episode
duration.
Next, we append to the portfolio value to our list of portfolio values.
Next, we check if the mode is train, at this point, we are finished playing all of our episodes,
so if the mode is train, then we'll save our neural network to a file called DeQuan.
Dot five will also save a scalar to Skellig PKU.
Lastly, for both train and test will save the rewards we got so the train data will be saved in train
NPI and the test data will be saved in test NPI.
All right, so now we're going to look at another script, which is just for convenience, for plotting
the rewards we saved, just so you don't have to write this yourself.
This can be found in plot.
R-AL rewards that pie.
Basically, it loads up the rewards, which you can switch to train or test depending on what you want
to plot.
And then it prints the average reward, the men and the max.
It also plots a histogram of the reward.
So we know the distribution of rewards over each episode.
Next, I'm going to show you how to run this, so when you want to run this in train mode and plot the
train results, you're going to run Python, RL Trader Torpy minus M train and then you can do Python
rewards that PI minus M train.
When you want to test, you just switch the mode to test.
So you do Python or outrated up pi minus M test and a python plot are all rewards that pi minus M a
test.
So here are the results for training.
The rewards are pretty good, we can even get up to a five X increase in our original investment, which
is extremely high.
For testing, they are also pretty good, our original investment usually always increases by a significant
amount.
By the way, remember that our data set is from daily closed prices in the span of five years since
the test period is half of the data set.
That means this increase is over a span of about 2.5 years.
So you can take this profit and calculate the effective yearly return to get a better idea of how this
agent performs compared to, say, the S&P 500.