In this lecture we are going to look at the code lab notebook that performs linear regression in order
to prove Moore's Law is true that the transistor count doubles approximately every two years.
This lecture is going to walk you through a prepared to call lab notebook although a very good exercise
which I always recommend is once you know how this is done to try and recreate it yourself with as few
references as possible.
As usual you can look at the title of the notebook to determine what notebook we are currently looking
at.
To start we're going to have all the same inputs as before with one additional import which is pandas.
Pandas is useful for working with CSB files which is the format our data happens to be in to obtain
the data we are going to use the w get method to get the CSB file hosted on my github
next.
We are going to load in the data.
The data doesn't contain any headers and it only contains two columns.
I would recommend verifying that this is the case.
The next thing we want to do is reshape x so that it's a two dimensional array of size end by one rather
than a one day array of length.
Then we also need to do the same thing for y in PI talk.
Although this generally doesn't extend to other libraries.
Remember that the convention is X is a 2D array of size and by D.
And why is a 2D array of size and by K where k is the number of outputs.
Next we are going to do a scatter plot of this data.
Notice that it shows exponential growth which we already know
so next we're going to take the log of Y which is what we discussed previously.
We see that we get back what looks like a good candidate for linear regression.
Next we are going to do a little bit of pre processing on the data.
Specifically we're going to standardize or normalize X and Y will want to keep these means and variances
around for later use.
So let's call them emacs for the mean of x as X for the standard deviation of X.
And similarly for y.
What we have to keep in mind is that by doing this the units of X and Y no longer make sense.
Previously X had units of years but this is no longer the case and Y had units of log counts which is
no longer the case.
We'll have to reverse the transformation later on if we want to get back to the original units
in the next plot.
We do a scatter plot again which on first glance looks exactly the same as it did before but if you
look closely you'll see that both the x and y axes are centered around zero and the range of values
is now small which is exactly what we want.
Next we're going to do a little more pre processing to convert our data into float 32
next.
It's time to create our linear regression in PI torch model.
Thanks to our rule all data is the same.
This code will be exactly the same as before.
Once we have our model which is just one line of code we're going to create our loss and optimizer as
you can see there's one argument here that might be new to you momentum.
This is one of those variations on gradient descent that I was talking about earlier.
If you want to know more about these you're encouraged to check out the ends up section of the course.
Next we're going to convert our num pi data arrays into torch tensor is using the from then pi function.
Next we're going to run our main training loop as promised.
This is the exact same thing as what we had earlier just with a different number of epochs.
I just want to mention again because a lot of people ask me how do I know what number of epochs are
needed.
Just remember that there is no way around it trial and error.
There was no magic formula.
You just have to keep trying until you're lost per iteration.
Looks right.
You'll be seeing the same loop over and over again throughout the course.
So each time I'm going to spend less and less time explaining it.
All right.
So from the loss values is printing out.
It looks like the loss has converged.
All right so next we're going to plot the loss per iteration as usual.
And from this we can confirm that our loss has converged.
Next we're going to plot our line of best fit.
Again this is the same as before so there's no need to explain this code again.
As you can see our line is indeed a line of best fit
next.
Let's get the trained weights of the model.
You already know how to do this.
It's model weight data.
Number pi as expected.
It's a two dimensional array that holds a single number.
The question is what does this number even mean.
To answer that question we need a little theory
let's start by recalling our model of exponential growth.
We have C equals c not.
Times are to the party as you recall we convert this into a linear equation by taking the log of both
sides.
So now we have log C equals log of C not plus log our times t.
If we rename the variables we can make this look a little more familiar
so let's call the input x the output Y and the slope a good thing to remember is that the slope of this
line is the log of are not r itself.
We can leave the intercept term untouched because it's not related to the rate of growth.
It's just the initial value of the count.
Next we have to take into account that our data is normalized.
As you recall we normalize the data using the equations we see here.
Let's call the transform data ex prime and Y prime.
So why prime is equal to Y minus m y divided by s y and x prime is equal to x minus m x divided by s
x.
Now we can begin to understand our own model our own model is also just align just with different units
and therefore a different slope and intercept so let's call the parameters of this line w and B so the
line that we actually fitted would be y prime equals w times X prime plus b
what we can do next is recover our original model by substituting X crime and Y prime with their expressions
in terms of the original variables x and y.
So now we have Y minus m y divided by S Y is equal to w times x minus em x divided by s x plus B you
should be able to see intuitively that this is still a linear equation.
After doing some algebraic manipulation we can isolate y in terms of all the other variables.
If you don't understand how we got to this step I would strongly recommend trying this on paper by yourself.
Well we can see is that there is one term involving X and three other terms not involving x.
Clearly the three terms not involving X are equal to the intercept log of C not remember.
We don't care about that because that doesn't involve the rate of growth.
Instead what we care about is the slope.
Luckily it has a simple expression it's just w times x y divided by s x
we know that our original line has the form a x plus log of C dot and therefore we can conclude that
if we had fitted a line to our original data the slope would be a equals w times s y divided by as x
all right.
So the next question we want to ask is what is the value of it.
Luckily we know w we know s y and we know s x so we can just plug them in to find a remember that W
is a two dimensional array holding a single value.
So in order to retrieve his value we have to index w twice at rows 0 in column 0.
We see that we get another sort of strange number zero point three four.
It's not clear how this would lead to a rate of growth.
That means the transistor count doubles every two years.
Of course that's because there is still more work to do
so let's go through a quick calculation to review once again our model for exponential growth is the
count c is equal to C not.
Times are to the party C not as the count when T equals zero and r is the rate of growth.
If we take the log of both sides we get an equation that's a linear in t
we can express this in terms of our usual linear regression variables Y equals X plus intercept.
This shows us that Y is equal to log C A is equal a log r x is just the time t and the intercept is
log of C not because a is equal to log R we can just take the exponential of both sides to get that
R is equal to each of the power a.
From here we can plug in the value for the slope that we found to get that are as equal to one point
for 0 7 although as you'll see we won't need to use this value directly
next.
Let's consider the doubling time or in other words how long it takes for C to double.
We can do this by thinking of the original equation as the original time and the original count.
This origin is completely arbitrary and as you'll see very shortly its values don't actually matter.
So in order to double this count the left hand side would become 2 C and on the right hand side the
time would just be t prime.
We call it t prime because we don't know what it is.
Yeah that's what we're trying to solve for
so if we take these two equations and divide one by the other we see that the C disappears the C not
also cancels out and as I promise the origin point doesn't matter we get to is equal to answer the power
t prime minus t t prime minus t is actually the time duration that we care about.
That's how long it takes for the transistor count to double how far from T is t prime we can rearrange
this to get that T prime minus t is equal to log 2 divided by log R which is equal to log 2 over a since
we found a.
This is just a number.
And more importantly it's a constant number one important note is that the defining characteristic of
exponential growth is that the growth rate does not change over time.
In other words the value of t itself doesn't matter the same equation holds no matter what the value
of t is.
This is like saying the duration from t to t prime is always constant and as you can see here it's always
log 2 divided by a
so in the next block of code we can print out this value log 2 divided by a is approximately equal to
2 just as Moore's Law says it should be.
Therefore we've confirmed that Moore's Law is true
in the final block for this notebook.
I have a little exercise for you in this lecture.
One of the most difficult parts was transforming the data using standardization and then reversing that
transformation and you might ask yourself Is that really necessary.
So the exercise for you is to find out what happens if you don't try to normalize the data.
You might assume that since the data falls on a line it should be pretty easy to apply linear regression
as we've done multiple times so far.
So give that a try and I'll see you in the next lecture.