In this lecture we are going to look at a modified version of the recommenders code.
This notebook will make some major improvements on the previous code and also I will talk about how
I discovered what modifications to make.
This lecture is going to walk you through a prepared call lab notebook.
Although a very good exercise which I always recommend is once you know how this is done to try and
recreate it yourself with as few references as possible as usual you can look at the title of the notebook
to determine what notebook we are currently looking at.
Okay so before we look at the code I want to make a special note about why it was rewritten and why
it might confuse a lot of beginners.
So from a general software engineering perspective everything we did in the old script was totally right.
We were using the functions that pi to work had given us and we were doing things the PI torch way.
One thing I really advocate against is when students come to me and say hey lazy programmer Why did
you write your code this way.
Why aren't you using this built in function.
Instead of writing this complicated code that's hard for me to understand.
Why are you reinventing the wheel.
And if you're a beginner or maybe you don't have that much experience yet but you are a newly minted
software engineer who is very intent on following the rules and being a good well-behaved software engineer.
Then you would tend to agree but I hope to teach you why this is a bad idea and I hope that you will
learn that being a good well-behaved software engineer is not always optimal.
So first of all if you're taking a course because you're trying to learn how something works then you
should expect to implement some things yourself.
That's what learning is.
As always my rule goes if you can't implement it then you don't understand it.
Okay.
So maybe it's hard for you to understand but this is not a valid reason for you to not do something.
If it were easy then you wouldn't need to take a course on how to do it in the first place.
If it were easy you should be able to figure it out yourself.
So if you're looking for things to be easy then you're probably taking the wrong approach.
I'm not a fan of programmers who excel at Reading documentation but are poor at problem solving.
Programming isn't an exercise in memorization but rather it's a task that involves building and putting
things together from basic building blocks it's better to ask how can I build this rather than asking
how do I spell some function name that does this one is a spelling exercise and one is a thinking exercise.
Some people say hey you're the lazy programmer aren't you supposed to advocate the lazy approach.
And to that I say being lazy is good.
If you're that good kind of lazy good lazy is when you're being more efficient.
Bad lazy is when you're trying to be more efficient but that leads to you being less effective and less
skilled and less knowledgeable.
So keep these ideas in mind as we go through the lecture
okay.
So in this script since it's so similar to the previous one I'm just going to go over the things that
are different so let's go down to where we define the model
now it's pretty obvious that I've trained this model many many times in the past.
In fact I have an entire course on recommender systems so I noticed that when I implemented this model
aside from the fact that it was training very slowly it was also not getting the means squared error
I expected.
So you might wonder how do I debug something like that if I know my tensor flow model performs very
well.
But my PI towards model does not.
What can I do.
Well here's one thing you can do.
First you want to make sure that both models are equivalent.
So if I create both models side by side and I copy the weights from one model to the other I should
get the same model predictions given the same inputs.
If I don't then something is wrong so you check that and it's a good sanity check.
And let's say that it works.
The next thing to ask is Well maybe the optimizes are implemented a little differently or they have
different hyper parameters.
This is totally possible.
So you can test that as well.
Basically you write your own custom training loop and for both models look at the data at the same time
and do the exact same updates.
If both optimizes are the same then both the last per iteration and the model parameters should be the
same after each update.
So you can plot the last pre iteration for both models and if they're the same then that means the optimizer
is the same.
So that's a good sanity check as well.
Of course before you start the training process you have to make sure that both models start with the
exact same weights.
Luckily you already know how to copy weights from one model to the other since that was your first sanity
check.
Okay so let's say you do that in the last pre iteration is the same for both models.
Now what.
Well you have to ask are you getting the worse mean squared error or the better means square.
If you copy the PI torch weights to the tensor flow model and both the optimizes are the same then you
should see the worst means quit air.
If you copy the tensor flow weights to the PI torch model and both optimizes the same then you should
see the better it means quit air.
So at this point you're narrowing down the source of the problem.
Basically it has to do with weight initialization
This is a topic that's outside the scope of this course.
So if you're not familiar with it please don't worry about it.
You can learn more in my in-depth series but basically there are different kinds of weight initialization.
For example you might sample from a uniform distribution or a Gaussian distribution.
From there you can choose the limits on the uniform distribution or the variance of the gaussian sometimes
programmers will make this depending on the dimensionality of the input or the output.
Well long story short if you plot a histogram of the initial weights of your PI to which model and your
tensor flow model you'll notice that they are not the same.
In other words that's the source of the discrepancy is that pi talk initialize as their weights different
leave tensor flow
luckily we have only two different kinds of layers in our network embedding and dense layers.
So what we have to figure out is where are these differences coming from.
How does pi torque initialize those layers by default.
And how does tensor flow initialize those layers by default.
And if they're both different do they both matter.
Or maybe it's that only one of them matters.
Lucky for you I've done all this work.
So what you get after doing a little digging is that although both the linear layers and the embedding
layers are initialized differently between pi torch intensive flow it's really the embedding layer that
makes a huge difference
specifically if you check the PI torch documentation you'll see that pi torch initialize is the embedding
matrix to come from the standard normal regardless of the dimensionality.
It's actually very surprising that this makes a huge difference since as you know we like to standardize
our inputs before putting them into a neural network.
Well if you sample from n 0 1 then our inputs will be exactly standardised.
So it's kind of surprising that this leads to poor results.
In any case what I ended up doing was manually initializing the embedding layers to come from a normal
distribution with a standard deviation of my choice.
So this is how you would do that.
So for each of the embedding we can access the weight attribute which then has a data attribute which
we can set simply by using the equals operation on the right side.
We create a parameter object which wraps a tensor object which wraps a nun pie array and that none higher
rate is a random matrix I generated myself all right so that takes care of one part which is the accuracy
of the model but we also have this problem that the model trains extremely slow so you'll notice that
in the script there are no more data loaders or data set objects
so we just go straight to the training function which takes in to new inputs the train data and the
test data.
These are just tools that store the NUM pi array versions of the data and if you've taken my in-depth
deep learning courses then you're in luck because we've written this loop many times for and in CNN's
Oren and so forth and back in the old days you would actually have to write this kind of thing yourself.
And by the old days I really mean like two or three years ago.
So I don't think what I'm doing is some kind of ancient secret or something.
This is still in the Deep Learning era.
OK.
So basically we're going to iterate through the batches manually.
Now most of this is the same but some of it is a little different.
So I'll walk you through the different parts.
And by the way if you already have some idea of how you would do this yourself you should try to do
that before looking at this.
So one thing that we need to do is calculate how long the inner loop will even be.
It's going to be the number of batches but how many batches do we have.
Well that's roughly the number of samples divided by the batch size.
Of course they may not divide evenly.
So we need to take the ceiling and then cast the result to an integer
so inside the main training loop we first shuffle the data.
We do this on every iteration to avoid unwanted correlations.
Then we do our inner loop through each batch inside the loop.
We grab the current batch.
The index range J times batch size up to J plus 1 times backsides.
You want to verify this for yourself by writing it out on paper.
Next we convert the batch to tenses and move the data to the GP you and we do our usual gradient descent
step
in down below we do a similar loop over the test set since this also may be too large to fit into memory.
You should confirm that for yourself
OK.
So next we call our training function after splitting up the data.
This is almost the same as what we did earlier except that this is in pure num pi and we only need to
convert the tensor when we're inside the training function.
All right.
So here are the results.
So if we look at the output from our training function we notice two things.
First the accuracy is much better.
These results are on par with what you would get with tensor flow.
But note that tensor flow works right out of the box whereas you had to do some customization in PI
talk to achieve the same results.
Second we notice that the duration for epoch is much lower than before before it took over five minutes
per epoch.
And this takes just under one minute per epoch so that's over a five x speed up just from reading our
own training loop and not using the built in data sets or data loaders which is very interesting.
So let that be a lesson to you.
Don't blindly take the approach of everyone must use libraries in order to be a proper software engineer
and don't reinvent the wheel.
Most people who say these kinds of things do so because they can't cope and because they can't code
they don't want anyone else acting like they can code.
They want everyone else to be just like them to appear that they have the same skill level.
So don't be like these people instead know how to implement things yourself know how to investigate
and debug and things like that.
Otherwise you'll just be a slave to APIs and libraries.