In this lecture we are going to define the environment class we're going to call it multi stock UniFi
at the top here.
We just have some comments describing both the state and action since these are non-trivial.
So as you recall the state consists of the number of shares of each stock that we own.
The price of each stock and the cash that we own on invested.
The actions can be different combinations of sell hold or buy for the three different stocks.
Next we have the Constructor it takes in two arguments a two dimensional data array containing a time
series of stock prices and also the initial investment amount with a default value of twenty thousand
dollars.
Next we set the attribute stock price history to the data we passed in.
Next we set the attributes and step and in stock which we obtained from the shape of the data array.
Next we set the attribute initial investment to the initial investment we passed in.
Next we initialize a few attributes to none.
Although we'll be selling these shortly next we set the attribute action space which is equal to the
integers 0 up to a 3 to the power and stock
next for convenience will want to have a corresponding action list.
Since we have twenty seven possible actions this will be a list of length twenty seven.
So the action space only tells us the integers from zero up to twenty six whereas for this thing it
actually gives us some examples of what we can have.
So for example 0 0 0 means sell all your stock 0 0 1 means sell the first 2 stocks but do nothing for
the third star.
Now this code may look complicated but you can try it in your terminal or in a notebook to confirm that
this is the result.
So you're gonna get an array consisting of values just like these.
Next we set that each state dim which will tell us the size of the state vector.
As mentioned earlier this is the number of stocks.
Times two plus one.
Finally we call the reset function which initialize is a few more attributes and returns the initial
state
so let's look at the reset function.
Here we set the attribute current step equal to zero.
This means that we point to the first day of stock prices in our dataset.
Next we set the attribute stock owned to an array of all zeros.
This array will tell us how many shares of each stock we owe.
So when we start we're not going to own any stock.
Next we have the stock price which tells us the current price of each stock on the current day.
This is of course just the time series index by the current step.
Next we have the cash in hand attribute which to begin with is our initial investment.
Since we've not yet bought any stocks.
Finally we want to return the state vector which is done by using the function to get ups.
We'll see what's inside this function shortly
next we have the step function which performs an action in the environment and returns the next state
and reward among other things.
First we check that the action that we passed in exists interaction space.
Next we get the current value of our portfolio and assign this to the variable we've thou.
Next we increment the current step pointer and also the current stock prices attribute so the day goes
up the next day and the price goes up to the price for the next day.
Next we perform the trade.
This is another function we'll be discussing later.
But you can assume it involves some combination of buying selling or holding of the stocks we are considering.
Next we get the value again and assign this to the curve our variable.
Next we can calculate the reward which is just the difference between the current value of our portfolio
and the value of our portfolio.
Before making the trade Max we said the Don flag which is true only if we've reached the end of the
time series MAX WE SET THE INFO dictionary which allows us to return at the current value of our portfolio
since this is part of neither the state nor the reward.
Lastly we returned the next day the reward done flag an info dictionary just like the open AGM API
so that's it for the public functions of this class.
Next let's look at the functions we made use of earlier.
First we have get OBS This returns the state by the way sometimes like right now we are going to use
the terms state and observation interchangeably.
There are cases where we might not want to do this such as when the state is some transformation of
the observation or it could even be multiple past observations stacked up together.
For this script however we'll assume that they are one in the same as mentioned earlier.
This is going to be a vector with three components.
The first component is the number of shares of each stock that we own.
So that's Stockholm.
This should be a list of size three.
The second component is the value of the stock.
This should also be a list of size three.
Finally we add the cash in hand attribute to our observation at the last index and then we return our
observation.
Next we have the get Val function.
This returns the current value of our portfolio.
This is equal to the number of shares we own for each stock multiplied by the value of each stock.
Added to the cash we have
lastly we have the trade function which is probably the most complex function in this class to start.
We need to grab the action vector from our action list which is index by the action argument which is
just an integer.
So this is going to return us vector of size 3 which will tell us for each of the three stocks whether
to sell buy or hold.
So let's just do a simple example to explain this more clearly.
First that the action will be an integer from 0 up to twenty six and we're going to use that to tell
us which actions we actually need to perform for each stock.
So for example if we retrieve the vector to 1 0.
That means buy the first stock hold the second stock and sell the third stock.
Next since we want to sell everything we want to sell before we buy anything we have to figure out what
needs to be bought and what needs to be sold.
So we are going to create these two lists sell index and buy index.
These will contain the indices for the stocks that we want to buy or sell to populate these lists.
We're going to live through action back then we check if value is equal to zero.
If it's zero.
That means we want to sell this stock and if it's 2 That means we want to buy this stock.
Next we're going to sell all the stocks we want to sell.
As mentioned this problem is greatly simplified because it's all or nothing for each stock.
We want to sell.
We're going to sell every share we own.
So we lived through each index in the sell index list.
Then we increment our cash in hand to be the total value of the shares that we previously owned so that
would be stock price times the number of stocks sold and then we set the number of shares we own to
zero.
Next we have our buy loop.
Again this is extremely simplified.
We're just going to loop through each stock one at a time and buy one share of that stock until we run
out of cash.
This is of course a great simplification since let's say we have nine dollars but the share we want
to buy next is ten dollars.
We can't buy it but maybe the next stock is only five dollars so we could have afforded one share but
we would never see this.
Since we set the can buy flight to false and exit the loop.
Of course this only happens on the boundary of the stock.
So you're looking at stock 3 and going back to stock 1.
In any case inside the loop first we check if we have enough cash to afford a one share of the current
stock so cash in hand must be greater than the stock price.
Then we increment the stock owned by one and subtract the value of the stock from the cash in hand.