In this section of the course we are going to focus on a new topic sequence data although deep learning
initially became popular for being really great at working with images.
It was not long until Deep Learning also proved its dominance with sequences as well.
Some examples of that are text speech and even financial data such as stock returns.
In this lecture we are going to discuss why we should model sequences and also look at some examples
so let's start with the simplest kind of sequence data.
At least this is what most people think of when they think of a sequence that is a time series signal
a time series signal can be any continuous value measurement taken periodically.
For example a company's stock price is probably the go to example
Another famous example is the airline passengers data set.
This is actually more practical than you might imagine.
If you can forecast the number of passengers you will have on your airline you can do all sorts of things
such as making sure the airport has enough workers to handle all the passengers.
You can modify your prices to reflect demand.
In fact that's what people do.
Blueberry surge pricing is an example of that.
It could also help airlines decide where to put their resources and marketing actually forecasting airline
passengers is a surprisingly big deal.
There are all sorts of articles and papers written on this topic.
I just found out that China will soon surpass the United States as the world's largest aviation market.
Another example of Time series is weather tracking.
This is something we all probably make use of on a daily basis.
You don't want to go outside without your umbrella.
If it's going to rain.
One interesting fact about weather is that it's a dynamical system that probably sounds complicated
but some terms you probably have heard of are Chaos Theory and The Butterfly Effect.
This is that even if we have the exact deterministic equations to describe a weather system our forecast
will still become more and more incorrect.
The further into the future we try to predict that's pretty counterintuitive because you would think
if you have the exact equation for something then you should be able to calculate all the future values
precisely but due to the butterfly effect this is not actually true.
As the saying goes a butterfly flapping its wings in Tokyo can cause a tornado in America small in decisions
like numerical round off error and your computer will ultimately lead it to your weather forecast being
completely wrong eventually.
This is actually really relevant to us because when you think of Time series our own ends you automatically
think How can I use a powerful model like an aunt n to do time series forecasting
another great example is speech or audio for which the obvious application would be speech recognition.
Now I'm not sure how many of you are into audio engineering or music production but if you are then
you should definitely recognize this as a time series.
Just open up your favorite audio file in the open source program audacity and you'll see how it's just
a time series of sound amplitude.
So all in all time series are everywhere.
Another type of sequential data is text which of course deep learning also excels at.
The interesting thing about a text data is that it can be treated as a sequence but in machine learning.
You don't have to.
In the olden days of machine learning we couldn't deal with sequences very well.
And so we resorted to more naive models the typical way of doing this would be to create what is called
a bag of words feature vector
basically it works like this.
Suppose you're doing document classification so for example you take an email and you want to know is
it spam and not spam.
So what do you do.
Well first let's take a big long feature vector with length equal to the number of words in the English
language.
Then for each word that each component of the feature vector represents we insert the count of how many
times that word appeared in our email
so suppose for simplicity's sake.
There are only five words in the English language insurance.
Lone pickles backpack and football.
So now let's say our email contained the word insurance three times.
The word alone one time and the rest of the words zero times then our feature vector would be 3 1 0
0 0.
In this way we can convert every email in our dataset into a feature vector
and then we get back to the situation where we can say all data is the same as our input x.
We again have a table of numbers.
They just happen to represent word counts for the labels is just 0 and 1 for spam or not spam.
As usual you can run any binary classifier on this such as logistic regression or a neuron that we're
but there's a problem with these bag of words representations when we only use word counts we lose information
about the order of the words.
Here's a simple example of why it matters.
Consider the phrase dog toy.
Now consider the phrase toy dog.
Clearly these two phrases mean something completely different.
And yet a bag of words model would treat them the same if you want to learn about how to deal with text
that would be a major topic of the next section.
But this section of focus mostly on modeling continuous valued sequences of data
usually when we think of a sequence we think of something like what you see here.
This is just like how we started with linear regression in the simplest way to look at it is a line
of best fit.
This is the most basic thing we can do because we can see it so analogous leaf sequences we're going
to think of one dimensional sequences because we can see them.
And while this is technically a sequence we can extend this concept a little bit.
Recall that when we were working with a non sequential data like what you saw with linear regression
that in general the data was of shape and and by D.
And as the number of samples in these a number of features as usual even when we are doing one dimensional
linear regression we still use a two dimensional array to represent the input data.
We still say it's end by D.
It's just that these 1
so far in time series are sequential data.
Let's think about what the shape of the data should be.
First consider just a basic sequence like a stock.
How can we represent the length of the sequence.
Should it be n should it be D.
In fact it's not really either of these we don't want to consider each point in the sequence to be a
sample nor do we want each point in the sequence to be considered a feature as a side note.
We could do that but that's not general enough for our own ends.
So it's clear we need a new letter to represent the length of a sequence.
How about the letter Big T.
That seems pretty intuitive because the word time starts with a letter T so it makes sense.
If we're dealing with a time series or any other continuous valued signal that the length of the signal
should be denoted with a new letter T.
Considering what you just learned.
I'm going to tell you that the shape of our data when we're dealing with our own ends will have a shape
and a by t by D.
Let's break down what this means.
We know that and is the number of samples and these the number of features and you've just learned that
t is a number of steps in a sequence as you can see this is a three dimensional signal and is the first
dimension t is the second dimension and d is the third dimension.
When you're dealing with sequence data sometimes students can get confused about which thing is n which
thing is T and which thing is D.
So I think the best way to understand this is by example
let's suppose we want to model the path that people take to get to work so you can record data from
the G.P.S. in everyone's car.
In this scenario what is n what is T and what is D.
Well since we're modeling people's trips to work one sample will consist of one person's single trip
to work.
You might record some data over multiple days in which case that person could have multiple trips to
work that would count as multiple samples.
All right.
So what's D.
In this case D is probably two because the G.P.S. will give you both latitude and longitude measurements.
Okay so what s t t is the number of lat long measurements taken from the time the person embarks on
their journey to work until they arrive at work.
So let's say it takes a person 30 minutes to get to work and you record at lat long coordinates every
second in that case the number of steps you would have is 30 times 60 which is eighteen hundred.
So the length of your sequence t would be eighteen hundred
now you might be wondering doesn't it take each person a different amount of time to get to work and
indeed you would be correct.
Intensive flow and chorus we generally work with equal length sequences so just assume that this is
the case for now.
We'll discuss this in more detail later.
For now let's think of more examples
here's an example of where d equals 1.
So it's kind of like the simple case analogous to our one dimensional linear regression a simple scenario
for this would be stock prices a stock price is a one dimensional signal the vertical axis is price
and the horizontal axis is time.
In other words to have a signal where D is bigger than one we would have more measurements per unit
time for a single stock price.
We just have one the price.
So what would enemy in this case.
This is a complicated question actually but let's consider again as something simple.
Let's say we want to break up our stock price signal into windows of length 10 so our task might be
something like using 10 sequential measurements of the stock price.
Try to predict the next stock price.
In this case and would just be the total number of windows of length 10 that we have by the way this
should remind you of some of the convolution or arithmetic we did.
If your total sequence of stock prices has Sei length 100 and you want to know how many windows of length
10 can be taken from the sequence.
That's just one hundred at minus 10 plus one which is ninety one.
In other words if you have l consecutive measurements and you have a window of length t then the number
of windows is L minus T plus 1
of course with stock prices.
You might not want to measure just one stock.
There could be multiple stocks.
So here's an example with stock prices where D is bigger than one.
Imagine for example that you've taken measurements for 500 stocks just like the S&P 500.
Now you have the stock price for 500 different stocks over time.
That means the equals five hundred you can again artificially make a window of length T and then the
number of samples you have is just the number of these windows
here's another example which is somewhat futuristic.
Imagine you are working with Elon Musk at neural link and you record voltages from electrodes placed
in the brain.
In this scenario D is equal to the number of electrodes at each point in time you're going to have measured
these different voltages at different locations in the brain.
One of the early applications of brain computer interfaces was moving a mouse cursor on a screen and
typing.
In other words everything you need to use a computer.
So suppose we are trying to predict to the letter or number you are thinking of or want to type.
And this is based on the voltages recorded from your brain for one second.
As you think of that letter a number.
Now let's say measurements are taken from the electrodes at a sampling rate of one sample per milliseconds.
That's 1000 samples per second.
That means for one second in the time we make the recording of you thinking of the letter we've collected
1000 measurements at each electrode.
So t equals 1000 and as usual would be the number of letters that you thought of.
So if we did a test where you had to think of a series of letters and collected at ten thousand one
second recordings.
That means an equals ten thousand
one question you might be pondering is why is our data of shape.
And by T.
By D.
Why not.
And by D by T or some other order and in fact there is no particular reason it needs to be this way
except for the fact that all Python libraries from machine learning conform to the standard.
This is just like our tabular X data where we have end by D.
Or our image data where we have n by H by W by C..
This could just as easily be d by N or n by C by h by W.
In fact we've already seen that in PI talks in Vienna.
The convention is end by C by h by W..
So this stuff is not set in stone.
My goal in this course is to show you how we do things in PI torch in Python the general pattern is
that we always put an a first.
So that should go without saying.
There are certainly other libraries which by convention do not put end first the other convention is
that we put the number of features last.
So that's why we have end by D.
Where D is the number of features.
This also makes sense for images where we have end by h by W by C because C is the number of feature
maps which is the analogous version of features for image data.
And this also makes sense for sequence data because this D still represents the number of features
by the way.
One important thing to do is visualize the shapes of the data in your mind before where we had and by
these inputs we visualize that as a rectangle rectangles are two dimensional so that makes sense.
Now we have a three dimensional object so it is no longer a rectangle it's a box.
So when you hear me say and by t by D you should automatically think of a box without needing me to
tell you to do so.
This should be very helpful as you write your code and reason about how Arnold's work
as promised.
One thing we have to consider is what if we have a variable length sequences.
It's clear that this is a pretty common scenario.
Just think of sentences not all sentences have the same number of words some sentences have one or two
words and other sentences have one hundred words.
This is actually a very interesting question.
In the past I've written code that can handle sequences of variable length.
The problem is this gets complicated fast.
Another problem with this is that it means you'll be using inefficient data structures.
If you have an end by tea by the box where t is constant then we can store this in a single number higher
rate that is efficient because a num PI was written to be efficient.
But if T is variable and depends on which sample we are looking at we might use the notation t event
instead.
Since t depends on the sample n in this case we cannot store this in a single number higher rate.
Instead we would have to use a list which is inefficient.
On the other hand there are some inefficiencies with constant Length Sequences as well.
In Pittsburgh it's easiest to start with constant length sequences so that's what we'll be doing in
this course.
This is the case with other modern Deep Learning libraries as well such as tensor flowing characters.
If we do end up having variable length sequences as input what we can do is party sequence with zeros
so that they all have the same length as the longest sequence for example if your longest sequence has
one hundred words then even sentences with one or two words would be the first one or two words and
then 98 or 99.
No values as you can imagine.
This is going to take up a lot of unnecessary space and processing time because the neural network still
thinks your entire.
By t by the box is full of legitimate data simultaneously.
It does make coding ordinance much easier.
So it's a tradeoff.
Ultimately this is the way we do things in PI storage.
So unless you wanted to write your own custom code this will be our data format for sequences
luckily what pi torch does is a kind of compromise between constant length sequences and variable length
sequences.
The idea is instead of having all your sequences padded to be the same length as the longest sequence.
Just do that across single batches.
As you know our method of training in deep learning is batch gradient descent so we're only going to
look at a subset of our training data at any given step during that step.
We can pad the sequences in the batch on the fly so that they are only as long as the longest sequence
in that batch instead of the entire training set.
Now you might think this is still going to be a problem if you have one sequence in your batch that
has Length one hundred and all the rest are length one.
Then you'll have ninety nine padded values in all those shorter sequences but what pi torch does is
for certain generators.
It tries to grouped together sequences with similar sequence length so the batches you see will still
be random but not as random as they would be if you didn't care about the sequence length.
Of course if you're creating your own data generator you would have to write that kind of custom code
yourself.
Finally note that it's possible to take things one step further and actually deal with a variable length
sequences almost without dealing with padding.
This part of the lecture is optional and outside the scope of this course.
So don't feel like you have to understand this to progress through this section the idea with this is
that although your batch will be of size n by t by D.
Where t is the length of the longest sequence in the batch.
If we know the length of each sequence in the batch then we can be intelligent about which data we process.
Ultimately it looks kind of weird because your batch of data will still be updated tensor but then using
the lengths of each sequence you'll convert it into an unpaid add tensor.
Then you'll pass it through the orange end unit and get back in unpaid it output.
And finally you still have to reapply the padding and then pass your data through the rest of the neural
network.
Unfortunately doing this requires you to keep track of the lengths of each of your data sequences even
while shuffling and iterating over randomize batches.
This approach is more involved.
So at this time it's going to be considered to be outside the scope of this course.
All right.
So what are the main takeaways from this lecture.
Number one when you have sequence data that means your data isn't just a single feature vector but a
value or a set of values that can vary with time.
Number two lots of people think of times series which are one dimensional values that very with time
but we can also have d dimensional vectors that very with time.
Some examples of that are weather readings from multiple weather towers or brain signals from multiple
electrodes on the brain.
Number three in modern deep learning libraries you're limited to constant Length Sequences.
If you want to do better you can use constant like sequences per batch instead of the whole dataset.
If you want to do even better you can convert those constant length batches back into a variable length
sequences but that's outside the scope of this chorus.
Number 4 by convention will organize our sequence data to be of shape and by t by D where n is the number
of samples t is the sequence length and d the observation dimensionality.
You should always be visualizing these shapes in your mind which will help your understanding immensely.
But I want to prompt you to do so and therefore you'll just have to remember to do it yourself.