And the section on corrective measures, which I'll look at data augmentation.
In the ZDNet paper, the authors proposes data augmentation scheme where the simply crap nine patches
from each image at different locations with a quater size of the original image.
Now the first four patches contain four quarters of the image without overlapping.
While the other five batches are randomly cropped from the input image.
Then after that the mirror the patches so that they could double the training set.
So that sat.
Supposing we have this image right here, we're going to obtain this first patch.
You could see clearly that it contains this region right here.
Then we could also obtain this other patch, which happens to be this region, then another patch which
is this region around this, then another which is this.
So this first fall we ensure that they don't overlap, so they don't touch either each other.
Then for the rest five we just crop from any position in the image at random.
So we have this this notice that this now matches up with this or rather this overlaps with this.
This too also overlaps with this.
We also have this now that overlaps with it doesn't overlap with any and then we have no overlaps because
we have this point here.
We have it overlaps with this and overlaps with this.
Anyway, we could pick the rest five for random.
So there we go.
Now, once we have this, as you could see, we've put all this different segments of the image together
and this is how we double our data.
So for this image, we now have this new image which is created.
Now this process is repeated.
Same for the density map to generate a density map wherein this parts are been recreated, as we could
see right here.
And then for us to train our model so that we have this sizes being equal, we will just simply resize
this.
So we have this now.
Also note that we could define our model such that it takes an arbitrary size, so we could define our
model so that it takes a size and then generates an output eight times smaller than this image or this
input image size.
So this means instead of having this in X and Y, we just have known.
And then known.
Right here, knowing full well that with the input we'll get is going to be divided by it because we're
going to pass through this tool.
Max, pulling layers and the VG model right here.
Recall, we we had actually three, not two.
So let's do this.
Let's have our get base model summary so we could see our three max pulling layers.
There we go.
We have this one, this and this.
So clearly, no matter the input share we have here, we would always have an output which is eight
times smaller than the input.
And this means that we could take an inputs with arbitrary sizes like, say, 500, because in the case
where we actually recreate a dataset, we're going to have a new image which leaves from 768 1024 that's
height width to 507 six 768.
So this means that we could take an input of the shape and we could also take an input of the shape
since this doesn't.
Need the output to be fixed.
Now, let's run this.
And there we go.
See that?
Everywhere is known.
So we could pass in arbitrary values right here.
Now, that said, let's go.
I had to implement the data augmentation scheme described in the paper first.
Yeah, we have this get run, which permits us passing a minimum value, maximum value, and then generate
a random number in this range.
So we could take, for example, get rand of say, 0 to 30.
It gives us 19.
Then we have this in H in W, which is the same as in Y equals this.
Okay, so there we go.
We have run this.
Now we have the code for getting the different patches which we saw on the slides.
Recall that this first four patches right here, those first four first four patches at fixed positions
because we want to be sure that they don't overlap.
So that's it.
And then we have now the last five patches.
Right.
Uh, last five random patches from the image.
So we have this first for last five.
Uh, first for last five.
We're going to see why we're using this shortly.
So firstly, we have our first patch.
We'll decide to take the h main coordinate as this height divided by eight, and then the W mean coordinate
as the width divided by eight.
Now, note that if we have an image, we will have this the height dimension and then the width dimension.
So we could decide to have this.
This is the height and then there's the width.
So this is the origin.
Obviously we have this origin.
Then we go up to the height divided by eight.
So let's say we have this position for the width dimension.
We have the width divided by eight.
Suppose we have this position, so we see ourselves around this position right here.
Now what we do is we shift a certain number of steps to the right and then shift this a number of steps
to the left.
And there we go.
We have our patch.
Now, the way we get this number of steps to the right and to the left is by simply using the fact that
on the paper was stated that each of the patches have to be a part of the image.
So we shift by a quarter of the height to the downward and by quater of the width to the right.
And then that's how we obtain the patch simply.
So normally this should be a bit bigger.
So we should have this.
And that's it.
That's why we have this right here.
And then this is when we take divided by two, while the weight is divided by eight divided by eight,
though with divided by two, you are divided by two, the weight divided by two others to ensure that
they actually never meet.
Now, initially this height, Max and Max are considered to be zeros.
Then later on we do the addition, which we just spoke of, where we have this patch, one zero, the
patch one is this.
So we're taking this coordinate, which is this plus a quarter of the height than to complete the competition
for the patch one.
We have the max, which is.
The patch one one Does the W mean which way we calculate it right here.
And that's this weight divided by eight now, plus the weight divided by four.
And this is repeated for all this, four patches right here.
Now, the difference with those five or the patches is that they are gotten at random.
So we use the get round function, which will define already to obtain this mean values right here.
But then we ensure that this doesn't cross a certain limit because if we pick the values for this h
mean w mean to be around the borders, then it may happen that when computing the max w max we may have
to go out of the image.
So to avoid this, we make sure that this random values generated range from 0 to 3 quarters of the
height and three quarters of the width.
So that's why we do make this choice right here.
And similar to what we had seen already initially, this max values are zeros and then we simply go
ahead to add up the height divided by four and then the width divided by four to obtain the max and
the max.
Once this is done, we have this patches which takes in now each and every patch right here, a patch,
one patch two up to patch nine.
Then we shuffle this again so that when we creating our image, we are going to have something that
isn't resembling previous augmentations for the same image.
Now, the reason why we're doing the shuffling is mainly because of this four patches right here, because
this already is gotten randomly.
So for this file which is fixed, we just do the shuffling to change the positions.
Now, once we have this, we have this patches.
Now note that also what's interesting is that from this we could obtain the corresponding patches for
the density maps by simply dividing the patches by it since all the positions actually match up.
Given that we divide the image which is an input by it to obtain the density map.
Now we could run this and there we go.
Yeah, our patches.
We have the values for the H mean W mean Max, W max and so on and so forth.
Now if we take this off, we see that we always have c we always have the same values.
The first four values are always the same.
Let's take this now.
There we go.
We now have the coordinates for all the different patches.
After getting all the different patches, we can now read our image.
It's basically what we used to do in already, which was simply reading this image.
And then for the image patch, there's this list will create.
And right here we are going to store the actual image patches based on those coordinates into this image
patch right here.
So that's why we have image patch or the pen we take.
Now the image array, which is basically the image.
And then based on those dimensions of the patch or this hardness of the patches.
So if you take this patch, for example, let's take this one because it's the image.
If we take this patch, for example, what we'll do is we'll simply take the H mean W mean H max and
W max or rather h mean h max.
This isn't w w anymore.
This h mean h max W mean w max.
So we pick out this eye for this range of all the different patch corners.
So here we have the patch coordinates and then here we actually get these patches.
So this means, for example, that in this case we could have something like zero to say 272 So let's,
let's take this 0 to 192 and then this could be from 0 to 256.
So there we go.
Basically, that's what we're doing.
We're just picking out a portion of this image.
Let's take that back and we'll find now that we have this image patch, you see, we just converted
to tensor and then we redo the same process, but for the map patch now.
So we create a map patch where based on what we got from this integration map, we actually pick out
this patch from it.
Once that's done, we now have this final step where we define this new image.
So there's this new image now which will contain all the different patches which we've gathered throughout
this nine different points here.
So we're going to have this.
We'll define this in new.
You see, it takes a shapes.
We could do this calculation to obtain the shape and the case where we have an input of 768 by 1024
by three.
Now this is going to give us the shape simply because if we have one and two by 256 and that we have
to repeat this nine times, it simply means we are doing this high dimension.
Let's take this off.
We're doing this high dimension three times.
So it's just like doing this three times.
We're repeating this also three times.
So if you multiply this by three, multiplied by three, you obtain this, which happens to be three
quarters of the initial input.
Now, once we have this, we go ahead to create this new image which we call new.
Now, that said, suppose we have this new garden from our which is going to collect all those different
patches.
It means we have this nine different positions.
So there we go.
We have this.
I have that.
And then we have this right here now.
So we have each patch.
One, two, three, four, five, six, seven, eight, nine different patches to fit.
And based on the patches we have seen previously and to fit this now all of those we just simply pick,
for example, go from 0 to 192 and the height.
And then in the width we go from when I zero will go from zero to like in this case we go from 0 to
256.
So that's how we obtain this.
So this patch, we select this patch and we put in the value of the image patch, which we had seen
previously recorded as image patches.
Now there are nine different image patches which we want to fit into this, different boxes right here.
Now when I is zero we have the image patch zero, so we have the zero image patch which will be fitted
in here.
So that's it.
Now we go to we still at I equals zero, we move to this, we now shift the height.
Basically what we're doing is we're shifting the height.
So we now move to 192.
So we're going from 109 to 292 times two basically.
So we have 384.
So this is the next.
And then though, since I is zero, we still maintain this.
So basically we're still going from zero 2 to 56.
So now this is what we occupy.
And then when I when we move to this, I still zero.
We have two times this.
So we have 384 to 507 six.
And then we have this portion right here.
Now when I equals one, you will see that we still have from we still go from 0 to 192 for the height,
so the height from this to this.
But then the width now goes from 256.
So instead of starting from initial year, now we go from 256 to 2 56 to 56 plus 256.
So that's it, 512 now doing this and match.
Now with this we see that we fall in this box right here.
So basically repeat this.
We have this, we have this when I equals two, now we move to this, this and this.
So that's how we fill out all these different boxes.
And the same is repeated for the mapping.
Now that's understood.
We could run this.
And there we go.
We see that we now have this our image.
Let's run this again so we could see how it's modified.
Notice that it's going to change.
So you see.
There we go.
So we see that we have actually done what's expected as of getting this different patches and then forming
this new image, which will multiply our dataset size by two.
Then the last step now is just for the this image, or rather this mappings.
That's this Gaussian mapping which seen previously.
So our density maps we should just verify to ensure that this is the same as the input.
You see this head right here.
Let's start from the so we have a head.
So you notice that we have this right here.
If we go down, we see this hat, it matches up with this, This hat.
Yeah.
Matches up with this.
This other one here.
This, this is what I had right here.
Matches up with this.
See this one, this, this.
So that's it.
Now, at the center, notice that the center, there's no head.
So that's why you see the center looks empty right here.
And then maybe this this consider to be a head, because very close to a head, it is not very clear
from here.
Now, looking at this top, you see there are many people looking at this, you see, because there
are many people here, we have this different points.
Then we have this point here.
Let's look for a different image.
Let's take this one.
Run that.
Okay.
Now, this image is better because there's some portions where there is no head.
So this one, for example, is very empty.
This empty.
See, now this portion is where you have this head.
You see?
You have this.
Yeah.
See where there's so many heads?
See this and that.
So that's it.
That's how we multiply this by two.
Now, let's integrate this into our data generator.
Now, before integrating our data generator, let's note that we also define this new function.
Generate new, which takes in a patch.
Now know that this patch could be image patch or density map patch.
It takes on this high dimension with dimension know that the density map and the image will have different
dimensions.
So that's why we're taking this.
And also even the image could have different dimensions.
In the long run, we can have that data set with different heights and widths compared to what we have
now.
Now, if it's an image, it's true.
If it's not, we have this right here.
And the reason why we're doing this is obvious.
Once an image, we have this channel dimension.
When a map, we have no channel dimension.
So that's it.
Now we repeat the same steps by just making sure that what we have now here are those variables which
will vary depending on the input image right here.
Now, that has been understood.
Let's look at this integration in our data generator.
So yeah, we have the X and Y.
So X took this, Y took that, that was fine.
Now we define those, we get our patches using the get patches function, which we've defined already.
Once we get our patches, we go through the same process where we have the image patch and the map patches
which we actually obtain.
And then the next step is to apply the generate new and then get this image, this new image generated.
We resize it to get the original, the size of the the original size of the image.
Now we had to find this to be in X and in Y.
So basically we have here in H.
So here we have in H, in W, here we have in H divided by it and yeah, in W divided by it.
So that's fine.
Once this is done we stack the to.
Elements of our list, both for the inputs and for the outputs.
So we could run this now.
That's fine.
We are base model has been run already, so we could go on with that.
And also there is this point we didn't mention is that we took off the reshape now in favor of this
because with the reshape we are obliged to specify its outputs.
Whereas now with this we don't have to specify the outputs.
Given that, what we're just doing is we're taking off this last dimension, and to take that off,
it just suffices to do this.
That is because the all previous dimensions and then for the last dimension, take just the zero value.
And if this okay, we can now train C let's take this back or train.
We get in this error?
There should be.
Because yeah, when this generator who made this error of doing this resize based on the modified version
of the hide, know that we have modified the height to match up with a new image generator.
So this height had changed its number, the initial height.
So that's why.
Yeah, we should take X and then y the x.
Yeah, y.
So that's it.
Run this again.
And it works just fine.
We've gone out of memory.
We just have to restart.
Now know that depending on your computing power, restarting may not help.
So in that case, you may have to reduce the disk dimensions.
Right?
So you have to resize our inputs such that we have smaller inputs which could get into the GPUs memory.
Nonetheless, with this GPU, when we restart and clear our memory, we could actually train on the
inputs and the augmented inputs.
So let's run this.
Everything is going to work fine.
Now run our data generator on that.
Different models go ahead and train.
Now we see that trains, we've cleared our memory and or just there was now enough space for it to train
on this data set, which has been augmented after training for around 38 blocks, you have the results
we obtain.
Let's now pause the training and see what the kind of maps it generates.
So test for this training set data right here.
The density map generated is not bad, although the number of people predicted is not exactly the same
as this.
But the model is learning to generate this density maps correctly.
Now let's test on others say 245, which is what we had in the slides.
So we have 74 right here.
And obviously this will get better as we continue with training.
Now let's test on an image which is not in our dataset.
We'll test with this image.
We should call people to read the several test images which we used.
Basically, we run this and then we observe the density map, which is generated from the input image.
You see how it looks close to the input image we have.
And also note that this image doesn't look like a typical image from our training dataset.
So our model is learning quite well and generalizing even when the test set isn't looking as the training
set.
Now, also note that in practice you will have to, depending on the kind of problem you want to solve,
get as much data as possible.
So if you are having a say, a camera or CCTV which is placed in this kind of position, it's better
for you to train on more of this kind of data.
Whereas if your CCTV is place such that these images look more like this, then it's better to train
on this kind of data.
So it's very important to take note that the model isn't of great use if you don't pick out the right
data to train that model with.
So that's it.
Let's now count the number of people we have in this image.
Now, after segmenting this image into these different paths, we count this number of people seven
plus 29 plus 26 plus 20 plus 18, 14, 14, 14, 17, 28.
This gives us a total of 187 people.
And we see that our model did quite well because it predicted 180 people.
So we missed out on seven people.
And this is also normal because we have this pose right here, which for this person, for example,
she wasn't very clear and stuff like that.
So it's also important to note that in a case where you are going to be predicting people where they
are, polls like this, it's important to have as much data as possible where you have this kind of
post so that our model learns to differentiate between the polls and maybe the people's heads.
Like this poll seems to block this person's head right here.
So it's very important to take note of the dataset you're working with.
So it's very important to work with data set, which is representative of the kind of data your model
will be receiving once it has been deployed.