Before we move on it helps to think about what exactly we are trying to build.
Remember that eventually what we need to feed into the neural network is a batch of pairs of images
along with the target telling us whether each of those pairs is a match or not however.
Keep in mind that we do not actually want to store these batches of images because that would mean we
would repeat the same images and memory more times than we have to.
Instead it would be better if we could just generate these batches on the fly to that end.
What we actually won is just a list of pairs of indexes.
We'll have one array to carry the matching pairs and one array to carry the non matching pairs in both
arrays.
All we will store is simply a series of tuples of indexes so you can imagine if I have this big array
and the first two images the image is at index 0 and index 1.
If their matches I don't want to store those images together I'm just going to store 0 and 1 the tuple
and then grab those images later.
Now in order to create such a data structure it helps to first have a dictionary that maps from subject
to the index is where that subjects images are stored.
So that's what I'm creating here in the next block.
Train label to idea X and test label to idea X for both of these.
The key will be the subject tidy and the value will be a list containing all the indexes where that
subjects images are stored.
You should be able to see that this is what these two loops are doing.
I'm just looping through each of the train labels then I'm using the label as the key and adding the
index to a list which is stored as the corresponding value and then I do the same thing for the test
labels
in the next block is where I create my list of tuples for both train and test.
I'm going to have two lists one to store the matches which I'm calling the positives and one to store
the non matches which I'm calling the negatives.
It helps again to think of just a simple example.
So imagine we have just four images A and B which belong to subjects 1 and C and D which belong to subject
2.
For this example we will just have 2 positive samples A B and C so that would be stored in our positives
list for the negatives list we would have AC a d B.C. and b in total that 6 pairs which is equal to
4 choose to
so here's how we're going to build these lists in code.
We're going to start by looping over those labels to index mappings we just created.
Remember that the key is the label or subject idea.
Whereas the value is a list of indices where that subject's images are stored.
So the first thing I do in the loop is figure out all the indices that do not belong to the subject.
So I call that other indices so that's easy.
I just take the set difference between the full set of indices from 0 up to and train and then subtract
the set of businesses that belong to this subject.
And remember that this is not numerical subtraction but set difference in then I'm going to loop through
each index for each index that belongs to this user for each index.
I'm going to loop through every index after it in the same list.
So I live through illnesses and then I live through indices again but only after the index i and then
I append all these indexes to the train positives list
then I live through all the other indices and add all these pairs to the train negatives as a sign though
if you find this complicated I always recommend simply not following my code.
Instead write it yourself and you'll encounter all the obstacles and have to solve the same problems
on your own.
This will give you a much better understanding of what's going on rather than trying to follow me
and so after this we just do the same thing over the test set.
So it's the exact same structure except instead of a train label to index we have tests labeled to index
and we're populating test positives and test negatives rather than train positives and train negatives.