In this lecture we are going to continue looking at our Siamese network code specifically to take what
we just created and write some data generator functions.
These will be the functions that we iterate over inside our training loop.
Later on the basic idea is we're going to pre allocate some arrays and then fill them up with data from
the pairs of indices that we just created.
So we're going to start out by looking at the train and generator function
as you can see.
I've also specified the batch size here which will be 64 for the script inside the function.
I'm going to set and batches to be the number of batches it's essentially the number of positive samples
divided by the backside but rounded up since we round up.
That means the last batch will always have less samples than the rest.
Next we're going to enter an infinite loop inside the loop.
We're going to shuffle the positives so that's this one.
This is so that each epoch we'll encounter each of the positive samples in a different order.
There's no need to shuffle the negative samples since we are already going to be choosing those at random
next we're going to pre allocate some arrays.
First we set and samples to be batch size times too.
So in actuality the batch size will be 128 rather than 64.
And this is because the first half of the dataset will be the positive samples and the second half will
be the negative samples.
Next we create X batch 1 x batch 2 and Y batch.
Currently they are all just zeros and we're going to fill them up with an X for loop
so inside this loop we're going to do this in batches times.
First we get the indices from the positive pairs for this batch.
So that's from AI times batch size up to AI plus 1 at times back size.
Next we set the index J to 0.
So J will be used to index our patches as we add samples to the pre allocated arrays.
Next we loop through the pairs of positive samples for this batch so we get the X1 and RDX to
then inside the loop we grab the train images at ADX one and RDX 2 and add these 2 x batch 1 and x batch
to then we set y batch to 1.
Since these are all matches then we increment J.
So that on the next round we store the image in the next spot.
Next we enter another loop this time for the negative samples but before we enter this loop we choose
a random set of samples using NPD at random by choice.
The difference inside this loop is we set y batch to zero.
Since these are all non matches.
Finally the true data for this batch only goes up to the index J.
So we index X batch one X batch too and Y batch only up to J.
Now we have our batch of data but there are a few small issues.
First our images are currently of size n by h by W which is not inappropriate.
Input shape for CNN.
CNN wants its input to be four dimensional and in PI towards that end by color by height by with.
So we need to add a superfluous one dimension in between the batch index and the height index.
Additionally although we could do this in the training loop itself we can save some space by converting
our data into torture tensor is now.
As a side note even though Y is technically a binary integer label it's more useful to have it as a
flow because we'll be multiplying it in our cost function so from that we get x1 x 2 and Y and then
we yield x1 x 2 and Y.
Note that we must enclose X1 the next 2 with square brackets to make it a list.
So this will not work if you return a tuple and it will also not work.
If you try to just return x 1 x 2 and Y without any brackets this is because these two variables will
be the input into the model and Y will be the target
nice we have the test generator.
Now this is essentially the exact same thing except that we don't need to shuffle the data and it's
using the test variables rather than the train variables.