Welcome everyone to this section on Core Data Concepts, where we're going to be focusing on the measures
of data.
Now fundamentally at the core of using data science to solve real world problems and find solutions
to business challenges, we use data, and it's important to understand core concepts of data before
continuing on to use it with a variety of methods.
So in this section of the course, we're going to be focusing on some core topics to understanding what
data actually is.
And as you begin to learn about probability statistics and visualizations regarding data, we need to
take a little bit of time to understand what do we actually mean when we say the term data.
So the natural question arises what is data?
Formally speaking, we can think of data as collected observations and information about something which
can be structured or unstructured.
Let's think about a few examples of data to get an idea of what we mean by terms like collected information
or structured versus unstructured.
Imagine that you went to Antarctica and you started collecting information about penguins.
Maybe you take a penguin, make some measurements, and then send them back out into the wild.
Well, you can start organizing this information and maybe you give each penguin a unique ID and then
you also do measurements like the flipper length or the body mass in grams.
You should notice already the idea of units of measurements and also structure.
Here we can see a clear structure in a tabular form similar to a spreadsheet where you have columns
and rows, where each column indicates possibly a feature of this particular data set, and then each
row indicates an actual unique measurement of a particular penguin.
Now, often when we think about data, we're thinking about numbers.
But we should point out that not all data has to be numeric.
For example, we could also take note of the sex of the penguin, whether they are male or female.
Or you could take note of maybe the colour of a penguin's beak, whether it's yellow or orange or bright
red, etc..
So again, keep in mind that not all data needs to be numeric or even really have some sort of unit
of measurement.
I should point out, this is actually a real data set, and the data was collected and made available
by Dr. Gorman at the Palmer Station in Antarctica, which is a member of the long term Ecological research
Network.
And you can Google search penguin dataset and you can find more information on it.
So we've got an idea that data doesn't necessarily need to be numerical.
For example, the colors of cars or the sexes of those penguins, whether they're male or female.
And so we have this idea that there's different types of data.
And in fact, we can distinguish between different types of data, such as continuous versus discrete
data as well as structure data versus unstructured data.
Let's talk a little bit more about this vocabulary of data.
So in order to explore some core concepts, we need some vocabulary use to describe them.
So we're going to be talking about continuous versus discrete, sometimes known as continuous versus
categorical.
We'll also talk about structured versus unstructured data.
We'll talk about nominal versus ordinal data as well as population data versus sample data.
Let's begin by discussing continuous versus discrete data.
So it is discreet data.
Formally speaking, you'll often see discrete data described as it can only take certain values.
And I like to think of it as there's no values in between values.
So let's give you an example of this.
Let's imagine you're running a used car lot and you're taking inventory of your cars and you want to
take inventory of the car model.
Is it a Toyota?
Is it a Tesla?
Is it a Ferrari, etc.?
This is known as discrete data.
The car models can only take certain values and there's no real values in between values.
For example, there's no value that's really in between Toyota and Tesla.
The car model has to be one of those certain values, so it has to be a brand or model of a particular
car.
I should point out that you can also do this with playing card values that have some numbers in them.
For example, playing card values have to be ace, two, three, jack, Queen, King, and so on.
There's no playing card value that's 2.5 or in between certain values.
So don't get confused with discrete versus numeric data.
You can have numeric data that is also discrete data depending on the context.
So for example.
Let's imagine that you're marking down all the possible values of rolling a single die.
This is clearly numeric data, but it's still discrete because no matter how many times you roll this
die, you're not going to be able to get any other value besides one, two, three, four, five and
six.
So this is numeric data and it's also discrete data.
There is no 3.5 on the die.
Now let's talk about continuous data.
Formally speaking, continuous data can take any value.
That is to say there's going to be an infinite amount of values in between any two values if you're
able to get precise enough.
For example, the height or weight of people is a continuous value set.
So someone could be 172 centimetres tall or 173 centimetres tall.
And we can see this continuous data because there are values in between this that someone could be.
So someone could be 172.5 centimetres tall.
And if we're able to get super precise with this, someone could be 172.54 centimetres tall or 152.542
centimetres, etc. So hopefully that distinguishes the idea of continuous data versus discrete data.
And remember that while continuous data is numeric, such as a weight being 160 kilograms, discrete
data can be numeric like a dice roll of two or a string like the color blue.
You should also keep in mind that sometimes the context and framing of a data set will decide whether
you should think of data as continuous or discrete.
And in fact, when you're communicating with your colleagues about data, context is super important.
For example, let's imagine I just asked you, is color data continuous or discrete?
Based off what we just described, you may quickly say, oh, it's going to be discrete, like the color
of a car.
So you may just want immediately say there's discrete values.
Something is yellow, orange or red or green or blue, etc..
But if you start to think about this more like on a spectrum of wavelengths, then maybe the context
is more physics based.
So if we're in a physics lab and we're actually measuring the visible spectrum of light, then color
starts to become more like a continuous data set.
And in this context, but maybe color is not the best word, and I should actually be describing something
like wavelengths as a continuous data set.
Although technically, if we're only in the visible spectrum, I could use the word color.
That's why context is so important here.
So now we're really thinking more about wavelengths in nanometers.
And in fact, as I mentioned, does the term color even make sense here?
So there's gamma rays, x rays, radio waves, etc., which we can't see.
So keep in mind that context is super important.
And being careful with the vocabulary you choose with describing data is also going to be very important.
And also don't confuse numeric and ordered discrete data with continuous data.
Let me talk about an example here to clarify what I mean.
Let's consider an airline with passenger classes.
There's a first class, a second class and a third class.
Now, this is technically numeric data.
We could say it's one, two and three in the same way.
There's one, two, three, four, five, six on a dice roll.
However, there also is an order to this.
So second class is technically in between first and third class.
But why is this still discrete?
That's because there's no values that are any values that can be taken.
That is, there's only a certain amount of discrete values that the data can take.
It can only take first, second or third class.
There's no 1.57 passenger class.
So this is kind of an interesting example of it's technically numeric data because it's you could think
of it as one, two, three, and it also has a clear ordering, which kind of gives the presence or
feeling that some data sets are in between others.
But really it's still discrete because the data itself has to be one of these certain values.
It can't just be any value on a continuous spectrum.
So to help distinguish with this type of data, we need some more vocabulary, and that's where the
terms nominal versus ordinal come into play.
Nominal data is classified without a natural order or rank.
So, for example, if we're thinking of categories of the screen, animals like dogs, cats, lizards,
horses, etc. this is nominal data.
There's no real ranking that's official to these particular animals, although we may think dogs are
the best animals.
But that's for another time.
A good test for nominal data is to see if it can be clearly sorted or not.
Nominal data cannot be sorted.
So if you just gave someone a bunch of species of different animals, there's no real clear definition
for how they should be sorted or not.
Ordinal data can be sorted.
That is to say it has an order to it, that's all.
So that's a good, helpful reminder of what ordinal data is.
So our previous examples of passenger classes is an ordinal discrete data set.
So it's still discrete data, it's also numeric.
But the factor here is that it's ordinal.
So there is a natural ranking of these items first, second and third class.
So we understand that second class is in between first and third class.
I should also point out that ordinal data doesn't necessarily need to be numeric.
So whether data terms such as going to be hot, mild or cold, those can be said to be ordinal because
we can think of hot, mild and cold as being able to be sorted.
So from hottest to coldest temperature words.
When thinking about continuous versus this great and nominal versus ordinal.
Try to keep in mind the context of the problem you're trying to solve.
It may not be necessary to apply labels such as ordinal if they aren't useful to the challenge at hand
or the problem you're trying to solve.
Let's also talk about structured versus unstructured data.
So we need to understand that not all data is going to be formatted nicely in a table or spreadsheet.
We should also understand that in some cases you actually don't even want it in a structured format.
Structured data is highly specific and is stored in a predefined format, so this could be something
like an Excel spreadsheet.
JSON files, which are JavaScript object notation files, XML files, SQL databases.
They all follow some sort of predefined format.
That is to say that as long as you hand over the data to someone else who is familiar with the format,
they're going to be able to work with that data.
Then there's unstructured data that's not in any particular format, such as audio data or video or
text data that doesn't need to follow any particular predefined structured format.
So, for example, video data, that can be a short little YouTube clip or it can be a two hour movie
and there's no real self defined structure there.
It just happens to be in an encoding format.
So be careful not to confuse computer encoded file formats with formatted or structured data.
So just because a text is in a PDF format doesn't technically make it structured data because you'd
have to open up that PDF file to understand what you're actually looking at.
Are you looking at reviews or are you looking at some legal documents?
So unstructured data, again, is not in any particular format where you can just easily open it up
in a wide variety of programs?
So typically it's going to be that unstructured data is going to be harder to work with.
But in certain fields it's actually necessary to achieve results.
Many state of the art deep learning techniques use unstructured data to learn patterns and generate
new objects.
For example, you may have heard of Dolly two from Open Eye.
It's a deep learning program that is actually able to take custom text descriptions such as an astronaut
riding a horse in the style of Andy Warhol and then produce an image to reflect the text description.
Now, this is a super complex, deep learning model, but you get an idea that it actually used unstructured
data for its training set.
It used unstructured text descriptions as well as unstructured image files, and then was able to eventually
learn the relationship between a text description and the output image.
And after being trained on many, many images and many text descriptions, it was able to understand,
given a text description prompt, what sort of image should be output.
And I would highly encourage you to do a Google search for Dolly to to see all the amazing images people
have created with this amazing deep learning program.
And then let's also talk about population versus sample.
So finally, we also need to consider the scope of our data collection.
Is this data a full representation of everything available and known, or is it actually just a sample
of everything?
So population consists of every member of a group.
Keep in mind, this is dependent on the context of a situation.
For example, a list of all the student names inside of a school contains data on the entire population.
Because in this context, the scope of what we're looking for is just this particular school.
And if we get a list of all the student names in that school, that's the entire population here.
Context is super important.
Don't confuse the term population in this context with the population of an entire country.
In the context of data science, we use population to describe the entire available dataset.
Now, often it's not going to be possible to record data on an entire population.
In this case, we rely on a sample from the population, which is a subset of all the members of the
group.
For example, let's imagine we had an optional school survey that we only ask some of the students to
fill out in this context.
That would be a sample of the population of the entire school.
We should always try to have samples that are representative of the population we're trying to understand.
So having a survey where only two students fill it out in a school of 1000 students is probably not
a representative sample.
Later on, we'll discover that sample sizes are actually a very well studied science.
So, for example, how many students should we survey for a school of 1000 students to get a representative
sample?
You can intuitively feel that one or two students is not enough, but you also have an intuition that
it's probably not necessary to survey all 1000 students to get an idea of how students in the general
population feel.
So there's actually a really well defined science to what sample size you need to try to get a representative
sample.
So in case you're curious about the answer to that particular question, we'll discuss it later on.
But it actually depends a bit on our assumptions of the overall population and its distribution and
the task at hand.
You can find out more now at this particular Wikipedia link on sample size determination, or you can
just keep watching along the course and we'll revisit this idea later on.
So we've seen that data comes in different forms and that we need to be cognizant of the context surrounding
the data and more importantly on what we are using the data for.
The ability to measure certain features of data is crucial to understanding data sets, especially numeric
ones.
And I hope the vocabulary we taught you right now is going to be useful in describing your data sets
later on.
So let's continue on by exploring two key concepts of data measurements.
Measurements of central tendency like mean median and mode and measurements of dispersion, variance
and standard deviation.
We'll see you at the next lecture.