Hello, everyone, and welcome to this new and exciting session in which we are going to look at post
training quantization with TensorFlow.
In the previous session we looked at quantization, our training still with TensorFlow and now we'll
look at how to do quantization for a model which has already been trained.
Now, as you could see here, we have this Pre-trained model which obtains an accuracy of 84% and a
top five accuracy of or at a top to accuracy of 95.6%.
And then in the section, we'll quantile this model.
Check out on whether this quantized model occupies less space as compared to the original model, and
then also verify that not much model performance is lost.
Before we start with the quantization process, we should note that we are going to be using this TensorFlow
Lite library.
Now, this TensorFlow Lite library is a mobile library for deploying models on mobile devices, microcontrollers
and other edge devices.
So this means that in this environment where the compute resources are limited, it's important for
us to quantized the models since we will now get smaller and lighter models and also faster models.
Here we have a general overview of how this works.
You will see you pick a model like, for example, efficient net based model.
You convert this model to TensorFlow Lite using TensorFlow lite converter, which we are going to see
shortly.
We then deploy this by taking a compressed version or compressed TF lite file.
Now we already working with, for example, Keras files, which are HF five files.
Now this TF lite is some sort of compressed file, and this compressed file will be loaded in the environment
in which you'll be working.
And then from here we'll also quantized this from 32 floats to eight bit integers which can run on devices
with low compute resources.
So you earn the documentation we have here t of the light and you have tf light converter, basically
the Silverlight converter.
As the word goes, it's going to convert your models into the TF lite format and yours.
You could see these examples.
You have this tf like converter from a saved model, from a Keras model, from a function, from a model.
So let's say, for example, we work with the Keras model.
We just have the model past year and then you generate this TF lite model from this model making use
of the S.F. light converter.
Now, apart from this arguments, we also have the attributes.
So you could specify the optimizations the representative data set, which is very important in the
case where we're working with static quantization.
Remember that we study quantization.
We have to obtain the skill and zero point values by making use of unlabeled data.
So basically, as we had seen previously, all we need to do is to pass in the inputs, which in this
case are images.
And then this values will be inferred from the model's interaction with the inputs.
Then we have the target specifications, inference, input type, inference, output type, whether
to allow custom operations or not, and then whether to exclude the conversion method data or not.
So that's it.
That's our converter right here.
Now we pass out this code from the documentation, and then let's let's take this first year now apply
some of the attributes that we have this converter dot optimizations.
Let's get back to the documentation, the optimizations and here we could set this optimization with
this TF light optimize.
So let's open this up.
There we go.
You see, this takes different values.
We have default.
What I want to optimize for size, this is deprecated does the same as default, optimized for latency
does the same as default experiment.
This one is experimental, hence subject to change.
So what we're going to do here is simply take this default.
So that said, we have the t f the light to f the light, the optimize and default.
Just as we had in the documentation right here.
Now, we could also specify the inference input type and the inference output type.
So let's get back here.
We have.
There we go.
Converter, uh, inference input type is going to be unsigned int it and then let's copy paste this.
We have the output type which is going to be the same.
So that's it.
So we will specify this, the inference input type and the inference output type.
And now let's specify the representational data.
So this representative dataset right here, which in fact is a generator which permits us output the
input values because recall, we all we need in the static quantization is just this input.
So here we have the generator which yields the inputs and then here we just say converter dot representative
data set equals a representative data generator.
Yeah, we have our training data set Now we could take we could take all our training data set or just
a few.
We obviously don't need to take all the data set so we could just take like 20 and use that to obtain
the values for the skill and the zero point.
Now, if you new to this notion of scale and zero point, it's important to check our previous sessions
where we treat this.
So let's run this here.
We also run this and now we set to convert all this model years pre trained our pre-trained model.
So let's run that and that should be fine.
So we now set to carry out the conversion.
Now we're done with the conversion.
We are now going to save this in the TF lite format.
So we have this path and this path we are going to have this file, so let's run the cell and that's
fine.
So we get that we have the file size here and when we check this up, you see we have this 21 megabytes.
So we're going from this model which we could check out here.
We're going from this model, which is 90.7 megabytes to a 21.12 megabyte model.
Now, before moving on, we should note that if we want to implement dynamic quantization here, then
we wouldn't specify this representative data generator.
So that's it.
Let's get back.
We install this TensorFlow lite runtime.
Now, talking about installing the TensorFlow Lite runtime, once we already have this TensorFlow lite
file right here, if we want to run this in some other system, say for example, want to run this in
our Raspberry Pi, all we'll need to do now will be to install this runtime and I'll be it.
We wouldn't need to install TensorFlow any longer.
So we just have this.
We run this that gets installed, we import the flight runtime, then we prepare our test image.
So we just run this.
We've seen this already.
We have our Pre-trained model.
We're going to get the max and then the corresponding class.
So that's it.
We should get angry.
See, it matches with what we expect.
We could try out this other example here.
Let's run this.
And there we go.
So this our model.
Now we're going to use this runtime to run our TensorFlow lite model.
Now we've restarted the session and you'll see that without TensorFlow, let's let's let's do this.
Let's say TF Zeros.
And one by two, for example, we run that and you see this is not defined.
So we have no inputs for now.
Now let's take this off and then we get back up here.
We install our runtime in part the runtime.
Oh, I think we'll be needing non.
So what we'll do is we're going to take this nom pi.
So we we trying to work without necessarily needing TensorFlow.
So we have this imported as numpy.
That's it.
Here we have this test image open CV so we will import CV two.
That's fine.
Well here we, we make use of TensorFlow but we will see how to get.
Read of this dependence on TensorFlow.
So first of all, we have to note that TensorFlow was used here to convert this test image here.
Let's print out our test image to convert this test image, which is an unsigned int with it bit.
So your let's do this.
You see it's an unsigned into one to convert this into a float.
And that's why we made this change here.
So you wouldn't need this any longer.
And here we could use nom pi.
So we have that and everything looks fine.
Okay, so we have this test image and that's fine.
So now let's run this and we have our image.
Now, image is not defined.
Let's get back here.
No, this is the test image.
Test image.
Run that again.
This should be fine.
Okay, So we have now our image.
And then here we see we have this interpreter which loads our TensorFlow lite file, which will save
the drive, and then we allocate to answers.
Once this is done, we move on to get the details, the input and output details which we had from the
conversion process.
Now here we have the unsigned end and here we also have the unsigned int and then you see that we have
this test image which we will change the type.
So first of all, you notice that this image is ten to an umpire array, which we don't need any longer
because this already non pi and then even this type.
We do not really need to do this, although if you print this, let's command the section and if you
print this input details and we get the data type, you will see that we have an unsigned int Now to
have not defined well here we'll use a light.
So let's, let's have this run that again we have to have like runtime has no attribute interpreter.
Now what we'll do is we'll have this dot interpreter.
Okay, so we should have this and this should work now.
Okay, that's it.
Now we have this.
We run this again, and that's fine.
So you see, we have you see the unsigned int, which is what is expected, because when doing the conversion
we have specified that we wanted this data type for our input and our output.
So that's it.
Let's take this off.
What we're saying is we don't necessarily need this step right here, so this will be useful If we had
this as a as a tense of as a tensor TensorFlow tensor.
And if we do not have this as an unsigned int already.
So now that we have that, you see we set the tensor.
So here we have our test image, that's it.
And then the input details index, you could print this out so you see what's in your input details
index.
As you could see, it's zero though.
Now in this line we get in an error.
We got three but expected for for the input.
So let's get back here.
Oh okay.
Yeah.
We, yeah, we had this change to m Let's have we actually let's get back.
So what we're saying is here we have this expand DIMMs to get from three dimension to four dimensions
and the name was still M So let's have that.
We run this again here we have our test image now which has four dimensions and then now this should
work.
So here we set the tensor and then we run the inference.
So that's it.
We run the inference here.
And once we run the inference, we should be able to get the tensor at the level of the output.
So let's take this off and then run this now.
Takes a while.
Yeah.
This.
This is an inference process.
Now, it should be noted that TensorFlow lite has been built for mobile and embedded CPUs.
So general purpose CPUs like this collapsed CPUs are the best match for TensorFlow lite models in terms
of speed.
Now we print out the output of just let's print out the output, see what's what's in there.
See, there we go.
It shows that we have this here, the highest value.
Then we could do an arc, max, of this output right here.
We run that again.
That's it.
Now, let's do let's take this here so we could get the class name automatically.
Then, now let's run this and we get the class happy.
So this is what we expected.
And again, we've done this without having to import TensorFlow.
So we did not we didn't need TensorFlow once.
We already had our TF Lite model, because here we have not defined.
Now our next step will be to measure the accuracy of our TensorFlow lite quantized model.
So here we do exactly the same process as we had before.
We basically have the model path and then this input details, output details, and then we go to our
validation data set, take 100 elements.
Now validation data set.
So this means that we have to import TensorFlow for this process since we are trying to evaluate the
model's performance.
So here we have this.
We will need the validation data set, so we'll need to get back and run this cells here.
So let's get back here.
We will run this.
There we go.
We re running all the cells.
Now we want to import this.
When TensorFlow is already imported, we get in this error.
So what we'll do is we just have this year we run that, there we go.
And then when we get to this accuracy, you will notice we use that the TensorFlow lite.
So it's tf dot lite, not tf lite.
So we using this model from TensorFlow and not from this package which we had installed here.
So that said, now we have that set accuracy.
We have input output.
As we're saying, our validation date is set test image, which is going to be passed here.
The inference we get, the output we compare, if they're the same, we increase accuracy.
If not, we skip and then we move on to increase the total.
And then from here we have accuracy divided by total or let's, let's say positives divided by total
or let's say correct, correct your correct predictions.
So let's change this.
So correct.
So here we have correct predictions and that should be it.
So we have this accuracy here and then we specify the model path.
Let's get back here and then take this path of our TF Lite model.
Now we we have simply accuracy, accuracy, and then we specify that path.
Okay, let's have this and we run this now.
There we go.
We're done with computing the accuracy for our TF Lite model and we get 0.82.
That is 82% as compared to the 84% with the original model.
Now, to get the more accurate value for this, it's advisable to use the whole data set.
So you could take this off.
And so now you have your model which performs at 82% accuracy and which now could be deployed in some
mobile device.