In this video, we will apply Varma using stats models in Python.
We'll begin this lecture by updating stats models to make sure we have the current version.
The next step is to import everything we need for this notebook.
Most of this you've probably seen before except for VMAX Inva.
The next step is to download our data set, so this data set is a data set of temperatures in multiple
cities.
The next step is to load in our data set using pedigreed CSFI.
The next step is to call ahead to see what our data set looks like.
OK, so notice that this data is not formatted in a way that's easy for us to process.
The first column is just a record ID, which we probably don't need the next three columns or month,
day and year, which we would prefer to have as a single day time.
The next two columns are for the temperature and some measure of uncertainty.
The next three columns are for the city, countryside and country name, the final two columns are for
latitude and longitude, which we won't use, but you might find useful in other models.
OK, so before we do anything else, let's think about the data format that we would like to have.
As you recall, we like our multivariate time series to have the shape TBD, that is T. rows and columns.
For this example, we use vehicles to for simplicity, that is, will choose two cities.
Note that our dataset is not in this format since each city is actually a column itself.
So all the time series for each city are stacked on top of one another.
Now there are multiple ways to parse this data into the format we need.
I'll show you one way, but keep in mind that there are many ways to do this.
So the first thing we'll need to do is figure out how to pass the date to do this well, write a function
called pass date, which takes in a whole row of the data frame.
The idea is we're going to call the appli function on our data frame passing in this function.
Note that the appli function is analogous to the map function in Python.
Basically, it means apply the same function to every row.
So inside a function, we're going to use string interpolation.
We grab the year, month and day from the row and put them into a string with dashes to separate each
item.
The next step is to call the date time, a strip time function passing in our string along with the
corresponding format.
The next step is to call the appli function passing in the date function we just created will assign
the output to a column called Date.
The next step is to extract the temperatures for two different cities, you can see that have chosen
Auckland and Stockholm for no particular reason.
Feel free to choose whatever cities you like.
Note that there's probably no relationship between the temperatures in these two cities because they
are so far apart.
So we'll see what this means for our models predictive ability.
So to get this data, we say density equals equals than the city name.
We make a copy and then we call drop in to remove any missing values.
Now, recall that our data frame has a bunch of columns we're not going to use, that is data, which
is not the Time series itself.
So in this next block, we're going to select only the temperature and the date.
Now, we don't actually want to be a column.
Instead, we would like it to be the index.
However, in the previous data frame, the date was not unique, since you could have the same date
for multiple cities.
Now that we have only one city for our two new data frames, we can set the date as an index.
So we call the set index function passing in the date column.
The next step is to drop the date column since it's no longer needed.
The next step is to rename the remaining column.
Keep in mind that we'll need to join our two data frames together eventually.
So we would like each data frame to have a unique column name.
In this case, it's just the city name concatenated with the word temp.
The next step is to call the head method to see what our new data frame looks like.
OK, so we can confirm that it's now in the format of a Time series like the ones we've seen before.
The next step is to perform the same set of operations on our Stockholm data frame.
The next step is to call the head method again to confirm that what we've done is correct.
The next step is to join our two data frames together.
The next step is to check the shape of our new data frame.
So as you can see, it's about three thousand rows and two columns, however, note that we have some
missing data specifically at the start of the table.
Now, as you'll see very soon, a three thousand is actually a pretty large number of values for Obama
to handle.
So the next step will be to just select the last five hundred rows.
We'll call this joint part.
We'll also take this opportunity to set the index frequency, which is months.
The next step is to call it is in a function to check how many missing values remain.
So as you can see, we have one missing value in the first column and four missing values in the second
column, which is not too bad.
The next step is to call the interpolate function to fill in the missing values.
The next step is to call it is in a function again to confirm that the missing values have been filled
in.
OK, so since the sum of missing values is zero, we can confirm that the missing values are gone.
The next step is to plot our data, to see what it looks like.
OK, so notice that the temperatures for the two cities are not on the same scale, so perhaps it may
be useful to scale this data.
OK, so at this point, our data is now finally in the format we need, and we've also seen how it looks
since this lecture has been pretty long.
We'll stop now and continue in the next video.