Hi guys, and welcome to this new section in which we are going to be looking at quantization for neural
networks.
In the previous section, we looked at the open neural network exchange standard, which is an open
standard for machine learning interoperability.
We saw that not only does this Onnx format permit us to convert models from one framework to another,
but they also allow us to optimize our models for different hardwares.
And so in line with this optimizations, we are going to look at quantization, which is a technique
for performing computations and storing tensors at lower bitwidth than the usual floating points which
we have been working with so far in this course.
Model Quantization is a popular deep learning optimization method in which model data that is both the
network parameters and the activations are converted from a floating point representation to a lower
precision representation, typically using eight bit integers.
Now defining quantization in this manner may not seem very clear.
So let's try to understand first of all why quantization or quantizing a neural network model is important.
So here let's consider this very simplified model where we take in some input, multiply it by a weight
and add the bias.
Now we have several layers, so we just simply stack this up and we could say we have our model which
is already been trained and this model has 100 million parameters and occupies, let's say one gigabyte
of space.
So we have this model, 100 million parameters, one gigabyte of space.
And if you're doubting what this space is for, you should note that it's for storing this weights and
biases.
And so obviously the more parameters we are going to have, the heavier our final model file will be.
Now, supposing you want to use this in some setup like a mobile phone, so you want to use this in
your mobile phone.
It means that you will need to allocate at least one gigabyte of memory space if you want to run this
model.
And this is where the techniques like quantization come in.
So now, thanks to quantization, instead of storing this weights in a 32 bit space, we are going to
store them in eight bit memory space.
So going from floating point 32 to INT eight.
Now, if you're not familiar with the floating point arithmetic, you could check out this resource
by Fabian Sinclair where he explains in a very intuitive manner the whole concept around floating point
binary representation.
Essentially if a single weight value, that's your model weight, let's take this off.
If you have a model weight value, which is, for example, 3.14, the way this is represented in memory
is by, first of all, allocating this 32 spaces we have here where each space takes a zero or a one.
And this first position here, the zero or the one, is to specify whether we are dealing with a positive
or a negative number.
And then for the next eight positions, we are going to say whether this value 3.14 lies in the range,
two to the -1 to 2 to the 0 or 2 to the 0 to 2 to the 1 or 2.
To the 1 to 2 to the two, and so on and so forth.
Now, in our case, 3.14 lies in this range.
And given that this exponent we have here is one, we are going to apply this formula where we have
the exponent -127 should give us this power we have here.
So we'll have that one.
So we have E is equal now 128 and if you convert 128 to binary notation, you would obtain this right
here and now.
After encoding this integer position, the next step will be to encode this decimal value right here
and that will be the role of.
Of this 23 other positions.
Remember, this is only one box.
This is eight boxes.
And here we have 23 boxes for this eight.
We've seen that it helps us locate our number in this range, which we've seen already.
But for this other 23 boxes, we are going to suppose that since we have two to the 23 possibilities,
thus let's write this here, two to the 23 possibilities, which is actually 8.3 88,000,608 possibilities.
So we have 8 million possibilities here.
This simply means that for every given range, which we have seen here, for this range, this range,
this range, this up to the end, we are going to divide it into 8 million different parts.
And so if you see here, you see you have two to the power of one.
This is two to the power of two.
If you break this gap, if you break this year, this year, this gap into 8 million different parts,
or better still, if we consider that the distance to move from 2 to 4 is 8.388 million, then the finding
this 3.14 encoding this 0.14 right here, or let's just say encoding 3.14 will entail calculating the
distance from to right up to 3.14, knowing that this distance from 2 to 4 is 8.388 million.
We can now compute this distance by simply doing 3.14 minus two.
That is 1.14 divided by all this distance.
That's two.
So we find this and then multiply by the 8 million we have 4781506.
So now we shall convert this to binary and we obtain this here.
So once we obtain this, we then fill up all this 23 spaces right here.
And that's essentially how a number like this is stored in memory.
And so getting back here, if we have to store this, let's say 3.14, 3.14, which was previously stored
in this 32 box memory, and now we want to store it in an eight box memory.
Now we move from 1GB to 256MB.
Here we have 256 and our mobile phone will now need only 256MB of memory to run our model.
Now, it doesn't just suffice to say we are going to go from the floating point 32 to the INT eight.
We need to describe exactly how this is done.
And the way it's done is actually by a simple linear mapping where we shall start by defining two ranges
of values.
The first range is for the floating point values.
And as you could see here, the define negative, a max to a max.
Well, one good thing about deep learning models is most times your weight or your weight values lie
between -1 and 1.
And so getting back here, we could have here negative one.
So a max will be one and two negative.
A max is negative one and then a max is one.
So we go from -1 to 1.
And then.
If we want our output to be unsigned ints instead of going from -128 to 127, we shall go from zero
to.
255.
Now notice that a number of values we have between 0 and 255 is the same as number of values.
We have between -148 and 127.
But with the unsigned ints, all our values are positive.
So instead of the int, we have unsigned int eight.
And so at this point, our aim is to take values ranging between -1 and 1 and map them in the range
0 to 255.
And now we will use a simple linear function which has the form y equals to a X plus B.
Now our wire will be the output value.
So we'll have the let's call this X.
We'll call this x.
Quantized is equal to x floating value.
So this is the original value of the weight.
Let's, let's put this in blue.
We have the original value of the weight or the float value divided by a certain scale, plus a zero
point value.
We'll call this Z.
So simply one over S is equal A and B is equal to Z, then Y is x and x is x.
F.
So now our aim is to look for the value of S and Z such that when we have any value in this range,
we get its corresponding value in this other range.
The way we'll get s, let's have this the way we get s.
Is by doing X.
Float Max.
X float max.
You see that x float?
Max is simply a max minus x float min, which is in this case, negative, a max divided by x quantized
max.
Minus x quantized min.
Now, if we replace all this by the corresponding values we have here, we will have one minus negative
one divided by two, 55 -0.
So 255 -0.
This means we have two divided by 255.
That's our s, which is our scale.
And then Z are zero point is x max minus x f max divided by s x we just had here.
So if you replace again, we have x max which is 255 minus x f max, which is one.
So we have 255 minus one divided by 255.
That will give us 255 divided by two is essentially 127 120 7.5.
So that's what you get.
Now, the way you can look at this zero point is it's the quantized value we get when the floating value
is zero.
So when we convert, when we have zero here, we have zero on S, which is zero.
The quantized is the quantized value or the corresponding quantized value is equal to Z.
So that's why we call that the zero point.
And then S, which is the scale simply scales our inputs here as we go from this range of values to
this other range.
Now, you could take a simple example where you could leave from x f to x SR.
If you have negative one here, you see, you have, let's say X is equal negative one, that's x f,
let's suppose we have negative one, then divided by s we said s was 252 on 255.
Um, plus Z.
Z is 255 divided by two.
So this gives us zero, which makes sense as we go from -1 to -1 one to 0 to 55.
This means that this here are these boundary values should be.
Almost the same.
Now, let's take another example in the middle, let's say negative or let's say 0.3.
So we could have X here, which is 0.3 divided by two on to 55 plus 255 divided by two.
So in this case now.
We have a value of 165.
.75, which if we run up, we could have 166.
So essentially we're going from 0.3 to 166.
And apart from rounding the output as we've just done, we would also see that we could clip any outliers.
So in case we've decided to have a max or our x max to be one, and that it happens that we have a weight
or weight value which is more than this one, then the output, um, unsigned int will be 255.
So any value greater than this is going to take this value, any value less than this is going to take
this value of zero.
And so we've seen how this simple technique permits us, reduce our memory used in storing the weights.
Now it's logical that we're going to have a drop in the accuracy, because if you've trained a model,
for example, to have certain floating or certain weights which are actually floats and then you convert,
this floats into integers where you have some extra transformations like the rounded and the clipping.
Then you would expect to have a drop in the performance of the model.
Nonetheless, this huge gains in terms of memory are enough for us to sacrifice a bit of the accuracy
or more generally, the model's performance.
And that's it.
Apart from this model weight size, which is dropped or reduced, it should be noted that arithmetic
operations like multiplication and addition of our quantized integers can be carried out even much faster.
And so we not only have a model which occupies less space, but a model which is even much faster.
We have generally three ways of carrying out quantization.
The dynamic quantization.
Static, quantization and the quantization aware training.
Now, given that during the quantization process, the weights and the activations are stored at lower
bit widths.
In the case of the dynamic quantization, this quantization parameters does a scale on the zero point
which we've seen already for the activations are computed dynamically or on the fly.
And because this year have to be computed dynamically, there is an increase in the cost of inference.
So it will take a little bit more time to produce an output as compared to other methods like static
quantization.
Nonetheless, here we usually achieve higher accuracy compared to the static quantization methods.
For the static quantization method.
We first of all compute the quantization parameters using a much smaller data set, which we'll call
the calibration data.
So essentially we have our model and in here we have our different quantization parameters.
But instead of dynamically computing these different quantization parameters, we are going to pass
in some inputs and outputs, carry out several runs such that we are able to obtain the most appropriate
quantization parameters based on this data we've passed in.
And then now when we want to run or carry out inference, we do not need again to compute these parameters,
unlike with the dynamic quantization, where at inference time we always have to compute this quantization
parameters.
Here, we compute this quantization parameters before via calibration data.
And then now when we run an inference, we just pass in inputs and we already have the quantization
parameters set.
But the problem now with this method is that if this calibration is done poorly, then we would have,
um, low quality values for the scale and the zero point.
And so because of this, we will then have a lower accuracy as compared to the dynamic quantization
method.
That said, these two methods we've just seen are post quantization methods.
So dynamic and static is post quantization, meaning that we train the model of first in floating point
32 and then after training the model, we convert this model to one with weights and activations which
are unsigned ints.
Now sometimes the post-training quantization that is is not able to achieve acceptable task accuracy.
This is when you might consider using the quantization aware training that security.
The idea behind the quantization aware training is simple.
You can improve the accuracy of the quantized models if you include the quantization error in the training
phase.
So unlike Post-training quantization where we train the model first before Quantizing here the network
adapts to the quantized weights and activations during the training.
So as we were saying, we include a quantization error in the training loss by inserting fake quantization
operations into the training graph to simulate the quantization of data and parameters.
These operations are called fake.
That's fake quantization because the quantized data, but they immediately do quantize the data.
So the operations compute remains in float point precision.
That said, the post-training quantization is more popular than the quantization aware training method,
thanks to its simplicity as it doesn't involve the training pipeline.
The quantization aware training almost always produces better accuracy, and sometimes this is the only
acceptable method.
And that's it for this section in which we've looked at quantization of neural network weights and activations
to help reduce model size and also speed up computations.