In this lecture we are going to switch gears a bit and discuss the kind of data we'll be working with
next.
Previously we discussed topics related to the model itself how a neural net will get structured and
the different kinds of functions involved.
That's the model.
But just as important is the data where deep learning and knowing that books really shine is on unstructured
data such as images sound and text.
And so for the next code example we'll be moving on to classifying images.
Of course this presents a challenge.
Previously you learn my rule.
Machine learning is nothing but a geometry problem and all data is the same.
We looked at some pretty simple examples with tabular data such as predicting whether a student would
pass or fail an exam based on how many hours they spent studying and how many hours they spent playing
video games.
Another example was predicting salary based on factors such as years of experience and degree type.
The question now is if your input data X is an image.
How does that fit into this picture.
As usual we're going to start by doing the dumbest thing possible.
But first we have to understand how images are represented in a computer in the first place.
So if we just look at an image let's say the famous Lena image we can see that the image has two properties
height and width.
Therefore your first thought might be let's store this in a matrix.
The next question is what do we store in this matrix.
Well suppose I have a matrix A and I'm looking at the position.
AJ This refers to the eighth row in the J column of the image.
You might think OK so AIG should store the color that appears at this exact position of the image at
position.
RJ The next problem we have to consider is how do we store colors.
In fact there are quite a few ways to store colors and computers but in this course will only discuss
the most common scheme called RG B stands for red green blue.
Now I want you to think back to kindergarten.
Yes all the way back to kindergarten.
I bet you didn't know that what you learned there would come in handy now if you forgot what you learned
in kindergarten.
Well I'm not sure if I can help you.
You recall that there's this idea of primary colors primary colors or a set of colors from which we
can form other colors by combining them a common set of Primary Colors is red blue and yellow although
red green and blue also makes a set of primary colors.
So what do you get if you mix red and blue you get purple.
What do you get if he makes a blue and yellow you get green.
What happens if you mix red and yellow.
You get orange.
Now this one you might not know but if you mix all the primary colors together you get white in fact
from the primary colors it's possible to combine them in different proportions to get any color.
So how do we represent colors.
Well in fact it's not just a number.
It's a set of three numbers.
This set of three numbers tells us how much red how much green and how much blue.
By mixing these colors in different proportions we can get different colors.
So what does this tell us about how to represent images.
Well in addition to height and width we need another dimension always of size three which represents
the three components of color.
So our image will actually be a three dimensional tensor AIG.
Okay.
This is the component of the image representing the eighth row J column and the K of color
the next important thing we have to discuss about how images are stored in computers has to do with
quantized fashion physically.
We know that color is light and we measure it with light intensity.
It's a physical value in the real world and therefore it's pretty much continuous if it's continuous.
That means it has an infinite number of possible values at the same time.
We know that computers don't have infinite precision and even if they did we might not want to make
use of it because that would make our image files take up lots of space.
Well it just so happens that people have already figured out that eight bits is good enough eight bits
gives us two to the eight possible values which is equal to 256 possible values.
These are encoded with the numbers zero up to 255 so in total considering that we have the red green
and blue channels this gives us two to the power eight times due to the power eight times due to the
power.
Eight possible colors which is equal to about sixteen point eight million now.
Using this we can actually calculate how much space it would take to store a RAW image.
Suppose we have an image of size 500 by 500.
Then the number of bits it requires would be five hundred times five hundred times three times eight.
That's six million bits.
If we divide that by eight to get bites we get seven hundred fifty thousand bytes that's equal to about
seven hundred thirty two kilobytes which is almost one megabyte.
You might realize that this is quite large for a 500 by 500 image which is why we have image compression
algorithms such as jpeg that help us store image files in a much more compact format.
As a side note for those of you who are web developers this is where hex colors come from.
These also happen to use the same color scheme and quantification.
So we have three bytes with each byte representing the value for the red green and blue color channels.
Importantly one byte can be represented with two hex digits hex digits go from 0 up to 9 and then a
up to F.
So it's a base 16 at number system to base 16 numbers give us 256 possible values.
So that's why we need to base 16 numbers or 2 hex numbers to represent one by then a color which is
represented by 3 of these bytes would require 6 hex digits.
So that's why when you're recording an HD CML colors are represented by six digits where the digits
can be 0 up to 9 and a up to f.
Now we can simplify this a little bit if we have an image that does not have color.
We call these grayscale images because each pixel can only be black or white or something in between.
Typically we assign black the value of zero in white the value of 255 any value in between is a different
shades of grey and because we only need one value instead of three grayscale images are stored in two
dimensional arrays rather than three dimensional arrays
in map LA lib.
If you tried to use the I.M. show function to plot a grayscale image it will end up assigning its own
color scheme.
But note that this is not the true color of the image.
This is just a heat map where we assign one color such as red to the high values and then another color
such as Blue to the low values.
So that's why you have to pass in C map equals grey when you want to show a grayscale image.
The last thing I want to mention about how images are represented in computers is that sometimes we
don't use 8 bit unsigned integers from 0 to 255.
If you recall neural networks don't like values on such a large scale.
So for neuron that works it's more convenient to scale image values to be floating points between 0
and 1 0 represents no intensity and a 1 represents full intensity.
And yes the astute among you might notice that if we represent images as values between 0 and 1 these
are not standardized values because they are not centered around 0 there are always exceptions and deep
learning for images.
This representation is particularly convenient because it's possible to interpret these values as probabilities
as mention.
There are always exceptions and deep learning.
So here is another one.
One of the most famous neuron networks today is called the VEGF which is used for many computer vision
applications such as image classification transfer learning style transfer and object detection.
Fiji also previously won the image that contest in multiple categories which as you recall is a competition
held every year for research groups to present state of the art image recognition models.
The interesting thing about Fiji is that the original author is did not scale the input data but they
did shift it by its meaning across each colour channel.
Thus for visually images are centered around zero but the range of values is still 256
so far.
We've discussed how images are represented at a computer.
However this still begs the question how is an image represented as input into a neuron that we're if
you recall our input data is usually represented by a matrix X of size n by D where n is the number
of samples and d is the number of features but this doesn't make sense in our case if we have an image
sample.
How can it have the features and image itself is a three dimensional object.
Therefore it would seem that in order to represent a data set of images we would need to have a four
dimensional tensor of shape and by H by W by C where n is the number of samples.
H is the image height W is the image with and c is the number of colour channels which is always 3.
Now this is a little bit of foreshadowing since this is what we'll do with later but for now we are
going to do something simpler
to keep things simple.
Let's say we have a black and white image of size three by three a very tiny image.
Well we can just number each pixel in a zigzag fashion going from left to right and top to bottom.
So we have pixel one pixel two pixel three all the way to pixel nine in the bottom right corner.
Well if we want to represent this as a vector why not just line them up in a row and call it a vector.
So we have a vector of size 9 containing pixel 1 pixel 2 pixel 3 all the way up to pixel 9.
We call this process flattening
in this way.
We can still store all of our images in an end by the array here and as the number of samples and is
height times with times color each row of this matrix is still one sample and each sample just happens
to be a flat in the image.
The features of the image just happened to be the individual pixel values.
And so here we encounter again this idea that all data is the same.
We saw previously how we can represent all sorts of different tabular data as an end by the array from
breast cancer detection to Moore's Law and so on.
Now we can represent images in this format as well so it is a truly powerful concept.