In this lecture we are going to describe the environment we'll be working with in the following lectures.
Firstly because we'll be working with historical stock data.
This is a simulation.
Of course you would not want to try such an experiment with real money.
So really our job is to figure out how to build an environment object in code that simulates the stock
market
in general.
Here's how we can think of the API for an environment by the way.
If you're familiar with open gym then this is probably just review for you.
But it's good to go over this anyway.
So the idea is this.
First we are going to instantiate an environment object.
Then we are going to initialize a boolean done flag equal a false.
We are also going to call in VDI reset which puts us back into the starting position for this environment
and returns the initial state.
As a side note this state vector may not be at a good scale to pass into a neural network.
As you recall remember that we like to normalize data before passing it into a neural network or linear
regression so keep in mind that you can do this as an optional step then we are going to enter a loop
which only quits when done becomes true inside the loop.
We are going to choose an action to perform in the environment.
This might come from our agent but that's not a necessary detail at this stage because we are only thinking
about the API for the environment you could just as easily choose a random action although most likely
this will lead to a suboptimal reward.
The next step is to actually perform the action in the environment.
We do that by calling Ian out step and passing in the action as an argument this will return a few things.
First it returns the next state.
Second it returns the reward for arriving in the next state.
Third it returns a done flag to tell us whether or not the episode is over.
Finally it returns and info dictionary which can tell us additional information about the environment.
This one is not strictly necessary and in fact it's empty for many environments but for us we actually
going to populate the info dictionary to tell us the current value of our portfolio.
This is in part of the state but can be calculated from the state variables.
Thus it's easier to simply calculate it inside the environment and return it along with everything else.
Finally we assign the next day variable to the state variable in the case where on the next step the
agent needs to use the state to choose an action
so that's pretty simple and you'll find that in general no matter what environment you are looking at
it's going to have an API.
Just like what we saw the questions we really want to answer now are what should the state be and what
should the action be and what should the reward be.
The reason we need to discuss these is because there are an endless number of possibilities and complications.
We aren't necessarily going to have to simplify the problem a little bit but first let me explain to
you why this simplification is necessary
let's start with the state.
There are many things you could consider here.
First you can think of it exactly like a time series problem.
Look at the pattern of stock movements in the past and from that make a decision.
That's probably the first thing you and I would do when we decide if we're going to buy or sell a stock.
However there are other things to consider.
We also have to ask do we own enough cash to buy the stocks we want to buy and given the prices of existing
shares I own.
Is it worth it to sell them in order to get more cash to buy a different stock.
So in fact this can become a complex decision problem.
We are going to borrow some ideas from a paper called Practical deep reinforcement learning approach
for stock trading.
This approach used a more advanced reinforcement learning technique known as DDP but we can apply a
few of the ideas they proposed.
So here's how we're going to represent our state.
It will consist of three parts First we're going to record how many shares of each stock we own.
So for example if I'm looking at Apple Motorola and Starbucks this means I own three shares of Apple
five shares of Motorola and seven Shares of Starbucks second.
We're going to list out the current share price of each stock.
So this means Apple is trading at fifty dollars per share.
Motorola is trading at twenty dollars per share.
And Starbucks is trading at thirty dollars per share.
Finally the last value of the state is how much pure cash we have.
That's cash that's not invested in any stock which just sits there and doesn't gain any interest so
let's say we have one hundred dollars in cash then our total state vector will be 3 5 7 50 20 30 and
100 you should be able to confirm that if we have any stocks than the size of our state vector will
be to end plus one
next.
Let's consider the actions again if we consider the sheer amount of possibilities the action space would
be extremely large for any given stock.
I have three possible options.
I can sell I can buy or I can hold.
Which means do nothing.
Now you might think three is not bad.
But now remember we have three stocks to consider for each of these.
I can exercise any of the three options above.
So that gives me three to the power three possible actions or twenty seven actions.
For example my action vector maybe sell sell sell which means sell my Apple shares sell my Motorola
shares and sell my Starbucks shares or it might be buy sell hold which means by Apple shares sell Motorola
shares and do nothing with my Starbucks shares however this is still not the end of the story because
this doesn't say anything about how many shares to buy or sell.
If I own ten shares of a stock I can sell anywhere from zero to 10 of those shares.
Luckily we are going to simplify this problem a little bit.
So here's how we're going to treat actions in our example.
It's going to be extremely simplified compared to how things work in the real world but it's a decent
start.
First we're not going to consider any transaction costs.
For example if you buy shares using your bank's investing platform usually that would cost you ten dollars
or so.
For us it will be zero next when we sell.
We will always sell all of our shares for that stock.
So let's say we own 10 shares of Apple stock and we decide to sell.
That means we sell all 10 shares secondly when we buy we are going to buy as many shares as possible
for the stock we choose to buy.
Now you might wonder if I choose multiple stocks to buy and I want to buy as many as possible.
How can I do that.
Well it's kind of ambiguous.
You might think you want to choose the stocks in such a way that leads to using up as much cash as possible.
But in fact this is actually a hard problem known as the knapsack problem.
So what we're going to do is we're going to take a simple greedy approach lived through every stock
and buy one share of each stock and keep doing that in a loop until we run out of money.
Third we will also sell the stocks we want to sell before we buy anything that will leave us with more
cash that we can use to buy new stocks this may seem like a very cautious approach but in fact this
already leaves us with 27 possible actions which means our neural network will have to approximate 27
different values which is pretty large already and so an action in this environment is not just making
a single trade but rather it will involve doing all the steps in the specified order.
Finally we have the reward.
This one is simple.
The reward will just be the change in the value of our portfolio.
Now it's We're thinking about how well we calculate the value of our portfolio as an example suppose
we own 10 shares of Apple 5 shares a Motorola and 3 Shares of Starbucks.
The corresponding share prices are 50 dollars for Apple twenty dollars for Motorola and thirty dollars
for Starbucks.
Let's also suppose we have one hundred dollars in cash not invested in any stock.
Then the total value of our portfolio will be ten times 50 plus five times 20 plus three times 30 plus
100.
That's equal to seven hundred ninety dollars
in general.
If we store the shares we own in a vector called S and we store the corresponding share prices in an
array called P and we store the amount of cash we have in a variable called C then the total value of
our portfolio can be calculated as follows.
It is equal to the dot product of S&P plus C the reward then we'll just be the difference between these
two.
Comparing the most recent timestamp and the previous timestamp
to summarize this lecture let's recap the important points about the environment and its implementation.
First our environment will be an object that mimics the open AGM API so it will have functions like
reset and step which returns all the information we need to implement our reinforcement learning program.
Next for our environment we'll be considering three stocks Apple Motorola and Starbucks next.
Our state is a vector with three pieces of information.
First it includes the number of shares of each stock that we want to consider.
Second it includes the share price for each of those stocks.
Third it includes the amount of cash we have that's not invested in any stock.
Next our actions are a simplified subset of the large number of actions we can perform in the real world.
We also assume there are no transaction costs.
Simply put we have three options for each stock buy sell or hold.
We'll take an all or nothing approach where if we buy we're going to buy as many shares as possible
and if we sell we're going to sell all of the shares we own.
And these reactions can be applied in any combination for each stock we own.
So if we're considering three stocks then we'll have three to the power three possible actions you'll
notice that even with just three stocks and a much simplified action space we still have quite a large
number of actions so if we have any stocks we would have three to the power and possible actions.
It grows exponentially with the number of stocks we own.
So encoding the actions in this way will not scale.
If we have many stocks to consider finally the reward is just the change in value of our portfolio from
the previous steps to the current step.
The value of our portfolio is just the price of each stock we own.
Times the number of shares we own plus any on invested cash we have.