-: What Allison should know-
-: What is Internet anyway?
-: Internet is that massive computer network?
-: The one that's becoming really big now
-: What do you mean that's big?
How does one-
what do you write to it like mail?
-: No, A lot of people use it and communicate.
I guess they can communicate with NBC writers and producers.
Allison, can you explain what internet is?
(Beethoven's Symphony no. 5 starts)
-: How amazing is that?
Just over 20 years ago
people didn't even know what the internet was.
And today we can't even imagine our lives without it.
Welcome to the Deep Learning A to Z course.
My name is Kirill Eremenko
and along with the co-instructor Hadelin de Ponteves,
we're super excited to have you on board
and today we're going to give you a quick overview
of what deep learning it is
and why it's picking up right now.
So let's get started.
Why did we have a look at that clip
and what is this photo over here?
Well, that clip was from 1994.
This is a photo of computer from 1980
and the reason why we kind of delving
into history a little bit
is because neural networks along with deep learning
have been around for quite some time
and they've only started picking up now
and impacting the world right now.
But if you look back at the eighties you'll see
that even though they were invented
in the sixties and seventies,
they really caught onto a trend
or caught wind in the eighties.
So people started talking about them a lot.
There was a lot of research in that area
and everybody thought that deep learning
or neural networks were this new thing that
is going to impact the world,
that's going to change everything,
is gonna solve all the world problems.
And then it kind of slowly died off over the next decade.
And so what happened?
Why did the neural networks not survive
and not change the world?
Was it the reason for that
that they were just not good enough,
that they're, you know,
not that good at predicting things
and not that good at modeling
and basically just not a good invention?
Or is there another reason?
Well, actually there is another reason
and the reason is in front of us.
It's the fact that technology back then
was not up to the right standard
to facilitate neural networks.
In order for neural networks and deep learning
to work properly you need two things.
You need data and you need a lot of data
and you need processing power.
You need strong computers to process that data
and facilitate the neural networks.
So let's have a look at how data
or storage of data has evolved over the years
and then we'll look at how technology has evolved.
So here we've got three years, 1956, 1980, 2017.
How did storage look back in 1956?
Well, there's a hard drive and
that hard drive is only a five, wait for it,
megabyte hard drive.
That's 5 megabytes right there on the forklift.
The size of a small room,
that's a hard drive being transported
to another location on a plane.
And that is what storage looked like in the-
in 1956.
You had to pay-
a company had to pay two and a half thousand dollars
of those days dollars to rent that hard drive.
To rent it - not buy it- to rent it for one month.
In 1980, the situation improved a little bit.
So here we've got a 10 megabyte hard drive
for three and a half thousand dollars.
It's still very expensive and only 10 megabytes.
So that's like one photo these days.
And today in 2017, we've got a 256 gigabyte SSD card
for $150, which can fit on your finger.
And if you're watching this video a year later, or like
in 2019 or 2025, you're probably laughing to yourself
because by then you have even stronger storage capacity.
But nevertheless, the point stands.
So if we compare these across the board
and without even taking price and size into consideration
just the capacity of whatever was trending at the time.
So from 1956 to 1980, capacity increased about double
and then it increased about 25,600 times.
And the, you know, the length of the period
is not that different.
So from 1956 to 1980, 24 years
from 1980 to 2017, 37 years
so not that much of an increase in time
but a huge jump in technological progress.
And that stands to show that this is not a linear trend
this is an exponential growth in technology.
And if we add into take into account price and size,
it'll be in the millions of increase.
And here we actually have a chart on a logarithmic scale.
So if we plot the hard drive cost per gigabyte
you'll see that looks something like this.
We're very quickly approaching zero right now.
You can get storage on Dropbox and Google Drive
which doesn't cost you anything cloud storage.
And that's going to continue.
And in fact, over the years
this is going to go even further.
Right now, scientists are already looking into using DNA
for storage and right now it's quite expensive.
It costs $7,000 to synthesize two megabytes of data
and then another thou $2,000 to read it.
But that kind of reminds you
of this whole situation of the hard drive and the plane.
You know that this is gonna be mitigated
very, very quickly with this exponential curve.
10 to 10 years from now, 20 years from now
everybody's gonna be using DNA storage
if we go down this direction.
And here's some stats around that,
so you can explore this further.
Maybe pause this, pause the video if you want to
read a bit more about this, This is from nature.com.
And basically you can store all of the world's data
in just one kilo, one kilogram of DNA storage.
Or you can store about 1 billion terabytes
of data in one gram of DNA storage.
So that's just something to
to show how quickly we're progressing and
that this is why deep learning is picking up.
Now that we are finally at the stage
where we have enough data to train super cool
super sophisticated models.
Back then in the eighties
when it was first initially invented
it was just wasn't the case.
And the second thing we talked about is processing capacity.
So here we've got an exponential curve, again
on a log scale, it's not ideally portrayed here
but on the right you can see it's a log scale.
And this is how computers have been evolving.
So again, feel free to pause a slide.
This is called Moore's Law.
You've probably heard of it
how quickly the processing capacity
of computers has been evolving.
Right now we're somewhere over here
where an average computer you can buy for 1,000 bucks
thinks at the speed of the brain of a rat.
And by 2025 will be the speed of a human, or 2023.
And then by 2050 or 2045
it'll surpass all of the humans combined.
So basically we are entering the era of computers
that are extremely powerful
that can process things way faster than we can imagine.
And that is what is facilitating deep learning.
So all of this brings us to the question
what is deep learning?
What is this whole neural network situation?
What is going on?
What are we even talking about here?
And you've probably seen a picture of something like this,
so let's dive into it.
What is deep learning?
This gentleman over here, Geoffrey Hinton,
is known as the Godfather of Deep Learning
and he did research on deep learning in the eighties.
And he's done lots and lots of work.
Lots of research papers.
He's published in deep learning right now.
He works at Google.
So a lot of the things that we're gonna be talking
about actually come from Geoffrey Hinton
and you can see a lot-
He's got quite a few YouTube videos
he explains things really well.
So highly recommend checking them out.
And so the idea behind deep learning is
to look at the human brain and there's quite-
there's gonna be quite a bit of neuroscience coming up
in these tutorials,
and what we are trying to do here is to mimic
how the human brain operates.
And you know, we don't know that much.
We don't know everything about the human brain
but that little amount that we know
we want to mimic it and recreate it.
And why is that?
Well, because the human brain seems
to be one of the most powerful tools
on this planet for learning.
For learning, adapting skills and then applying them.
And if computers could copy that,
then we could just leverage
what natural selection has already decided for us.
All of those kind of algorithms
that it has decided are the best
we're just gonna leverage that.
Why reinvent the bicycle, right?
So let's see how this works.
Here we've got some neurons.
So these are neurons which are, have been smeared onto glass
and then have been looked at under a microscope
with some coloring.
And this is, you can see what they look like.
So they have like a body
they have these branches
and they have like tails and so on.
So you, you can see then
they have like nucleus inside, in the middle .
And that's basically what a neuron looks like.
In the human brain,
there's approximately 100 billion neurons altogether.
So these are individual neurons
these are actually motor neurons
because they're bigger, they're easier to see.
But nevertheless, there's a 100 billion neurons
in the human brain.
And each neuron is connected to as many as
about a thousand of its neighbors.
So to give you a picture,
this is what it looks like.
This is an actual dissection of the human brain
and this is the cerebellum,
which is this part of your brain at the back,
it is responsible for like motorics
and for, you know, keeping your balance
and some language capabilities and stuff like that.
So this is just to show how vast,
how many neurons there are.
There like billions and billions and billions
of neurons all connecting your brain.
It's not like we're talking
about five or 500 or a thousand or million.
There's billions of neurons in there.
And yeah, so that's
that's what we're going to be trying to recreate.
So how do we recreate this in a computer?
Well, we create an artificial structure called
an artificial neural net where we have nodes or neurons
and we're gonna have some neurons for input values.
So these are values that you
that you know about a certain situation.
So for instance, you're modeling something
you want to predict something
you always gonna have some input
something to start your predictions off.
And then that's called the input layer.
Then you have the output.
So that's a value that you want to predict
whether it's a price
whether it's is somebody going to leave the bank
or stay in the bank?
Is this a fraudulent transaction?
Is this a real transaction?
And so on.
So that's gonna be your output layer.
And in between we're going to have a hidden layer.
So as you could see in your brain,
you have so many neurons.
So some information is coming in
through your eyes, ears, nose.
So you basically your senses.
And then it's not just going right away to the output
where you have the result.
It's going through all of these billions and billions
and billions of neurons before it gets to the output.
And this is the whole concept behind it.
That we're going to model the brain,
so we need these hidden layers
that are there before the output.
So the input layers
neurons connected to hidden layer neurons
that hidden layer neurons are connected to output value.
And so this is, this is pretty cool
but what is this all about?
Where is the deep learning here?
Why is it called deep learning?
Is nothing deep in here.
Well this is kind of like an an option
which one might call shallow learning.
We- there isn't much indeed going on
but why is it called deep learning?
Well, because then we take this to the next level.
We separate it even further
and we have not just one hidden layer.
We have lots and lots and lots of hidden layers
and then we connect everything just like in the human brain.
We connect everything, interconnect everything.
And that's how the input values are
processed through all these hidden layers
just like in the human brain.
Then we have an output value
and now we're talking deep learning.
So that's what deep learning is all about
on a very abstract level.
In the further tutorials we're going to dissect
and dive deep into deep learning
and by the end of it
you will know what deep learning is all about
and you'll know how to apply it in your projects.
Super excited about this.
Can't wait to get started
and I look forward to seeing you on the next tutorial.
Until then, enjoy deep learning.