So interestingly although we are doing something like binary classification our model does not actually
output a binary decision.
It only outputs distances between matches and non matches.
Thus it's actually up to us to choose a threshold at which we consider two images to be a match or not.
Now you might think since we push the matches toward zero and the non matches to at least one then the
threshold should be in the middle like zero point five.
But in fact the threshold we choose is a hyper parameter.
It could be less than zero point five.
Or it could be more.
One thing you'll see is that we are going to get really good separation on the train set but not as
good on the test said What this tells us is that it may be worthwhile to have three data sets train
of validation and test.
And this way we could use the validation set to choose a threshold which is often what we use to evaluate
all other hyper parameters as well.
Then when we've chosen our hyper parameters we can test our model on the true test set before we start.
Let's remember that when we're working with PI torch we need to carry our arrays from num pi land to
PI torch land before making predictions and vice versa.
And so here's a convenience function called predict that does all that for us.
It takes in to num pi arrays of images x1 and x2.
Then inside the function it converts X1 next to the torch tenses and moves them to the GP you.
Next we pass X1 and next two into our model to get a prediction.
Bring it back to the CPSU and then bring it back to num by land.
Finally we call the flat in function since the model returns and end by one output.
But we just want a one dimensional array
so let's look at this code to calculate the accuracy of our model now.
You recall that with this particular dataset one inherent property of it is that we have imbalanced
classes.
There are many more non matches than matches.
Therefore accuracy is not a reliable indicator of how good the model is.
Instead we'll report the true positive rate and the true negative rate also known as the sensitivity
and specificity.
So that's one of the things we're going to do in this function get train accuracy as you can see it
accepts one argument the threshold which will be used to assign a classification decision.
We're also going to make a histogram plot of both the matches and non matches so that we can appropriately
choose a threshold must store the distances we find for the matches in a list called Positive distances
and we'll store the distances we find for the non matches analysts called negative distances
we initialize T.P. t n FP NF n to zero and we'll accumulate these as we loop through the data
so first we're going to initialize to arrays X batch 1 and x batch 2.
Note that in this case we want to loop through every sample we have not just sample from the non match
class like we did in the generator.
So that's why we have a custom loop here and we're not using the generators we defined earlier
inside the loop.
We grab a batch of indices so that's pours back in is easier
and you'll notice that we have two loops so there's one for the positive list and there's one for the
negatives list so the tuples we encounter right now are positive matches.
Next we fill up X batch 1 and x batch 2 so that's pretty simple it's the same as before.
Next we use our predict function to get the distances for each pair of images so that distances.
And then we convert these into a list and store it in our positive distances list.
Finally we check how many of these distances are less than the threshold.
That's how many true positives we have.
Then we check how many distances are greater than the threshold.
And that's how many false negatives we have.
Next we have the same loop but this time over the train and negatives list this loop is largely the
same.
We gather the batch of data we pass it to model dot predicts to get the distances but this time we add
the distances to the negative distances list any distances which are less than the threshold are now
false positives and he distances greater than the threshold.
Our true negatives
once we are outside the loop we are done counting.
So we can calculate the true positive rate and the true negative rate.
And finally we plot a histogram of the negative distances and positive distances now because there will
be many more values in the negative distances list.
Technically speaking this histogram should be much taller than the other one but we can pass in the
argument density equals true.
So that map plot liberal normalize the height of both histogram is essentially making them into probability
distributions.
Next we have a function.
Get test accuracy which does the exact same thing but over the test set
and next we have the results.
Note that this takes some time because we have to loop over all the data points and generate the data
on the fly.
So I've printed out the batch number on each iteration of the loop
once we're done.
You can see that we get very good separation between the positive distances and the negative distance
is all the positive distances are close to zero and all the negative distances are close to one or bigger
than one as they should be.
Theoretically we could choose a threshold that would give us 100 percent accuracy on the train set.
But as I said earlier you probably want to do that with the validation set instead.
Let's see why.
So now we're going to look at the function and get test accuracy and I've pre computed this threshold
of zero point eight
okay.
So you can see that we don't get as good separation between the positive distances and the negative
distances.
There is a bit of an overlap.
Nonetheless we still get above 90 percent for both the true positive rate and the true negative rate.
Keep in mind that the results here will be slightly different each time you run the script due to various
factors.
First it depends on the images that we chose for the train set and the test set.
It also depends on the neural networks.
Initial parameters hyper parameters and how long it was trained for.
So I've gotten results better than this but I've also gotten results worse than this.
So test it out on your own and see what you can get.