OK, so in this lecture, we are going to continue looking at our notebook.
Previously we defined our data and transformed it so that it's in the right format to be ingested by
US forecast.
In this lecture, we're going to focus on uploading this data to three.
Let's start by defining our bucket name and our region, as you recall, bucket names must be globally
unique, so you can't choose the same bucket name as me or anyone else in the world.
By the way, I have assumed that you know how to create a bucket inside of a U.S. and that you've already
done so.
If you haven't, then you can either use Boso three or the Adewusi console to get that done first.
If you do not know how to do this and you need help, please make a question on the Q&A for the region.
You should use the same region for all your services to minimize network traffic and latency and especially
costs.
So basically, just choose whichever region is close to you that you normally use.
The next step is to set your role red, as you recall, we created this in an earlier lecture.
In that lecture, I mentioned that you should copy down this value.
And so this is where that value is used.
The next step is to create an S3 client using boats of three.
So we call Bodsworth three day client passing in the string as three and also the region name we defined
earlier.
Optionally, you can also pass on your credentials at this point.
The next step is to set a few more variables, our time series frequency is daily, so we create a variable
called data set frequency and set that to D.
Our timestamp format is year, month, day.
And so we set a variable called time stamp format, setting it to what you see here.
We also set a name for our data set group, which I'm going to call it Daily Forecast Data Set Group.
So just to give you a picture of what we are doing, you know that our data can consist of many different
files.
So far, we have two one for the Target series and one for the related series.
But as you recall, we can also have item metadata and we can have different item IDs, all of which
can come from different files.
We can even have different files for different days.
So if you have some time series where you're uploading data every day from different log files, then
you would have a different file for every single day.
So a data set group is essentially a group of data sets, each of which can consist of one or more files
in S3.
The next step is to create more boats of three clients for the forecast service and the forecast query
service for some reason, these are two separate services.
As a side note, I'm going to assume that you're familiar with the Boto three documentation where you
can look up all the functions for each of the services we are using, or at least you know how to read
it.
If you're seeing it for the first time since, we're only going to be using a subset of features in
these lectures, you are strongly encouraged to check the documentation for yourself to see what's available.
The next step is to create a data set group using the function, create data set group.
Here we have two arguments domain and data set group name where we pass in the group name we previously
defined for domain.
I'm setting this to custom.
So what does this mean?
So domain is actually really interesting because it allows you to specify what you're trying to forecast.
Presumably Amazon has some secret domain knowledge that would be helpful for these different scenarios.
For example, it's possible to set this value to retail for retail demand forecasting.
You can also set the store Web traffic to estimate Web traffic.
So you see that this service is really geared towards businesses who typically have to deal with these
sorts of measurements.
Earlier, you'll recall that I said everything we do in Adewusi forecast is a job, so this means when
we call these functions, what we're really doing is sending some commands to a bunch of computers to
do a bunch of work, believe it or not, simply declaring that you have a data set group, take some
time.
The next step is to get the data set group Aaryn from the response of our last function call recall,
I said earlier that everything we do in RWC comes with an error.
So now we have an Aaren for our data set group.
Once we have that, we can call the function, describe data, say group.
This will bring back a dictionary of which the relevant keys are status creation time and last modification
time.
I encourage you to print out the whole dictionary for yourself just to see what it contains.
Now, because of run this entire notebook, the status is already active, but if you're running this
live, it will probably say create in progress for some time.
As you can see, the last time I ran this, it took about seven minutes.
OK, so the next step is to create a schema for our data set.
We'll start with the target data center.
As you can see, it contains three attributes, one for the timestamp, one for the target value and
one for the item ID.
The type for the timestamp is timestamp.
The target for the target value is flow and the type for the item ID is string.
The next step is to give our data set a name, since our target data consists of closed prices, we'll
call it closed prices.
The next step is to create our data set using the create data set function.
So just to remind you, we previously created a data set group and the data set group contains data
sets.
So now we are creating a data set.
So again, we specify the domain, which is custom.
We also specify the data set type, which is target time series.
We specify the data set name using the name we created earlier.
We also said the data set frequency, which is daily, and the schema which we were just looking at.
Again, this launches a job which can take several minutes to complete.
So in the next block of code, we grab the Aaryn for our target data set from the response to the create
data set function.
Next, we call the described data set function passing in the area and we just retrieved from this we
get back another dictionary where we can look at the status creation time and last modification time.
Note that the status update in progress, this is because after running more code further down in this
script, I scroll back up and down this block again.
So it says update in progress.
Since after running this, I modified the data set by adding data to it.
Otherwise, the status should stay active when it is ready.
The next step is to follow all the same steps for a related Time series.
I want to make a note here that this is actually optional.
If you want to create a forecast from the original Time series alone, that's perfectly fine.
There's no requirement to pass in related data.
So if you only have a time series, then you can just pass that in as your Target Time series.
You might want to try that on your own if you're curious about how that will perform.
So we'll start by defining our schema, note that this now has more attributes, but timestamp in item
ID must still be present.
The new attributes are what you expect, open, high, low in volume.
The reason I've appended the word value to these attribute names is because of the word opium is a reserved
word.
So it's not an acceptable attribute name.
The next step is to give our new data, set a name, since this data set is for the related Time series.
I'm going to call this a related data.
The next step is to call the create data set function again, but for new related time series.
So what's different here?
First, the data set type is different.
It now says related Time series instead of Target Time series.
And as expected, the data set, name and schema correspond to the related series.
And again, since this launch is a job, the next thing we have to do is check the status of our job.
Again, you can see that the last time I ran this, it said update in progress because I reran this
line after I launched a new job to add the actual data.
Otherwise, this should say active when it's complete.
The next step is to add the data sets we just defined to our data set group, which we created earlier.
The first thing we do is create a list of our data set trends.
From there, we call the function update data set group where we pass in the Aaryn for our data set
group and the Arendse for our data sets.
The next step is to upload our data to S3.
We'll start by creating an S3 resource for each of our servers, will access our bucket and then create
an object within three key equal to our CSFI filename.
Once we have the objects, we can call the upload file function to upload the Carvey's that we have
locally.
The next step is to define the paths to our data in S3.
This is just the bucket name.
Slash the CSP file name.
As a side note, although we're not doing it in this lecture, it's possible to have multiple files
as part of a data center in order to do that.
You put all your curves into the same folder in as three and then specify the path here as the path
to the folder instead of being a path to a file.
The only caveat is that the curves have to be directly in the folder, so you can't have nested folders.
For example, you can't have different files for each day and then have folders for the year, month
and day and then specify only the year folder.
The next step is to launch an import job, to import our data into a US forecast.
Like I said earlier, everything we do is a job.
So to do this, we call the create data set import job function.
The arguments to this function should be straightforward.
First, we give our import job a name which I've just said to our data set group name.
Next, we said the data set Aaren, which is the target data set.
And next we set the data source, which basically points to our three target path.
Notice we also pass in Iran for our role.
As you recall, our role is what allows us forecast to access as three.
Finally, we pass in the Times that four men.
OK, so, again, we have to wait for this job to finish from the response to the previous function,
we can retrieve the are for the import job using this area.
We call the described data set import job function.
Again, we check the status creation time and last modification time.
So if you're running this for the first time, it should say active within a few minutes, you can see
that it took about seven minutes the last time I ran in.
Next, we're going to do the exact same thing, but for the related Time series, so we're going to
call create data set import job again, but this time will pass in the Aaryn for the related data set.
And there's three path for the related data set.
OK, so, again, we wait for this job to finish and we can call the function described data set import
job in order to check the status of our job.
As you can see, it again, took about seven minutes to become active.