Hello, everyone, and welcome to this new and exciting session in which we are going to see how we'll
move from a TensorFlow model which occupies one gigabyte of space to an onyx quantized model occupying
just 83 megabytes.
At this point, we now understand the concept of quantization and we're going to see how to apply or
implement quantization, specifically dynamic quantization to make use of our model even more efficiently.
Before we move on, we should also note here that the TF size TF size is one gigabyte is about approximately
one gigabyte.
That's 1000 megabytes, while the onyx size is 328 megabytes.
So onyx size 328 megabytes.
Now we're going to look at the Onyx Onyx quantized.
Let's just copy this.
So we have the Onyx quantized CPU, onyx quantized CPU, which we shall get shortly, the Onyx quantized
GPU.
And then we'll also get the onyx quantized size.
So let's take this off for now and then get back here.
Now, as you could see, we've imported the Onyx Onyx runtime quantization.
We've already imported the Onyx runtime.
So here we just imported this quantized dynamic and quant type.
Then from here you see we have the two models, thus the floating point and the quantized model.
Now, this year this model is what we had already.
So we'll get back and then copy that and then we'll call this one the white quantized.
So that's it.
Now all we need to do is we have this quantized dynamic which takes in this model, takes in the quantized.
The path to the quantized model is still an Onyx model and then the weight type.
Now here the suite type is an unsigned int.
So let's run this and see what we get.
We get an error.
This name is undefined.
Okay.
We should run this before running this one.
So that's it.
This should be fine this time around.
And we should be able to get this file right here, this quantized Onix file again.
Yeah.
You could check out the documentation on quantization on x models.
You see, we have the quantized Onix model.
We have an overview and all of that.
So let's get back here and check out our model.
Oh, there we go.
We have our bit quantized and what we notice is we have 83 megabytes.
So this means that we've gone from one gigabyte or 1000 megabyte to just 83 megabytes.
And now we could go ahead and check out the speed or the CPU speed.
So let's get back here.
Or rather, let's let's get back here.
There we go.
Copy this path.
Let's copy this path.
And then we have this year we paste that out here and then this this one year.
This isn't cuda this.
This is CPU CPU execution provider.
That's fine.
Everything looks fine.
And let's run this.
You see, we get 0.39.
That's practically 0.4 seconds per prediction.
And so here we have 0.4 seconds.
Now let's let's check this out again.
Let's check this out again for the original Onix model.
So let's run this and check this out.
You see here we have 0.49.
That's practically 0.5.
So it means that this isn't exactly true.
There should be 0.5.
So we see that the quantized model is faster and we much lighter than the Onyx, the original Onyx and
the TensorFlow models.
But we have to be careful as quantization generally comes with a drop in the accuracy.
Now we switch to a GPU and then test out our quantized model.
So right here we have this quantized model which we're going to run and then check out its latency.
Here is where we get 0.27, let's see, 0.3.
And this shows that the quantized model doesn't benefit as much as the Onyx model from the usage of
the GPUs.
Now, if you check out in the documentation, we'll see that there is quantization, an Onyx model,
and this is a quantization on a GPU.
So the quantization and the GPU isn't as straightforward as that with the CPU as here, we know that
we'll need a device that supports tensor core in a computation like the T four or the E 100.
And this here, that older hardware would not benefit from quantization.
So if you want to proceed with quantization or quantization with a GPU, you can make use of this tensor
R2 execution provider.
And here they give the overall procedure to leverage this tensor t execution provider.
So with that, we're going to get back here and the next thing we shall do is ensure that the QUANTIZATION
process hasn't led to too much drop in accuracy as when we quantized the model generally, we may have
drop in accuracy, but our Mir is to be sure that this drop is minimal.
And so to do this, we're going to evaluate our model.
So here basically we've defined this accuracy, which takes the model.
And then for 100 we'll take 100 elements, 100 elements in a validation data set where the validation
dataset has a batch size of one.
We are going to compare each time the output or the next prediction with the label.
So here we compare the level with the earnings prediction and if they are the same, we increase the
accuracy variable value by one.
Initially, they are zero total accuracy zero, but the total is always increased and then the accuracy
is increased only when we have this to the same.
So with this basically we implement this accuracy method, which now we take the two models, that is
the Onyx, the original Onyx and the quantized Onyx.
So here we have this providers to, to, to make this run faster.
So we run this.
Now you see here we have 90% for the original and then 89% for the quantized model.
The next thing we'll look at will be how to visualize Onyx models using Lute's rotors neutron app so
you could get your natural or app and you you have this interface right here.
Now we're going to open the model.
There we go.
It's loading and is what we get.
You see we start with this transpose.
You could recall we had this transpose and then we moved on to this resizing, then matrix multiplication
and then the rest of the model right here.
So here is the white model in this Onyx format, Onyx quantized format, screwed right to the end right
here.
And then towards this end, you see we have this matrix multiplications for a linear layer and then
we have this soft max.
You could also export as PNG.
So you could open this up in this PNG format right here.
So that's our model.
And that's it for the section where we've left from a one gigabyte model to an 83 megabyte model with
just 0.01 drop in accuracy.