1 00:00:00,400 --> 00:00:00,810 Oh, yeah. 2 00:00:00,870 --> 00:00:04,350 So I hope you enjoyed those lessons on convolutional neural networks. 3 00:00:04,890 --> 00:00:07,770 However, we're taking a step, a bit of a step back now. 4 00:00:08,190 --> 00:00:12,960 We're going to take a look at the history of deep learning, and it's a very important reason why I'm 5 00:00:12,960 --> 00:00:19,440 doing this for because right now we are basically on the treadmill on the cusp of a revolution. 6 00:00:19,950 --> 00:00:22,350 That's that's because of deep learning. 7 00:00:22,380 --> 00:00:23,220 And I'll tell you why. 8 00:00:24,000 --> 00:00:29,070 Firstly, yes, now is definitely the best time to become a deep learning project practitioner. 9 00:00:29,520 --> 00:00:31,500 We're in the golden age of AI. 10 00:00:31,830 --> 00:00:37,500 There's never been a time right now where things are moving so fast and everything is coming together. 11 00:00:38,010 --> 00:00:44,010 I've actually had this discussion in a few meetups and with lots of other people and in Big Tech companies 12 00:00:44,010 --> 00:00:48,690 like Google and Facebook and in video, and most of them share this view. 13 00:00:49,320 --> 00:00:51,750 It basically is. 14 00:00:51,750 --> 00:00:55,590 A graph will show you the next two slides, so I'll talk a bit about more about it. 15 00:00:55,590 --> 00:00:55,900 Do. 16 00:00:56,460 --> 00:01:03,630 But a lot of people underestimate the potential of a it kind of goes through a cycle where people overestimate 17 00:01:03,630 --> 00:01:04,770 and people underestimate. 18 00:01:05,160 --> 00:01:10,920 And is a reason for that is because continuously disappointed in the past, but something is changing 19 00:01:10,920 --> 00:01:17,040 now and that something is related to both deep learning and the accessibility of faster GPUs. 20 00:01:17,460 --> 00:01:22,830 GPU hardware, which is required to train these deep learning models, which is otherwise going on the 21 00:01:22,830 --> 00:01:24,960 CPU, is just is not feasible. 22 00:01:25,590 --> 00:01:33,360 It's GPUs have become so much faster thanks to in video meanly not not to bash in AMD, but AMD doesn't 23 00:01:33,360 --> 00:01:34,350 have CUDA support. 24 00:01:34,770 --> 00:01:41,040 And that's basically the low level library that all these deep learning libraries are built on. 25 00:01:41,430 --> 00:01:43,470 Now AMD is developing something. 26 00:01:44,310 --> 00:01:45,210 I think it's Vulkan. 27 00:01:45,690 --> 00:01:49,290 I'm not entirely sure much about it, but it's coming. 28 00:01:50,040 --> 00:01:51,780 So AMD is not going to be left behind. 29 00:01:52,260 --> 00:01:58,650 However, in video basically has a monopoly on this market right now, and GPUs are even free right 30 00:01:58,650 --> 00:02:04,020 now because as you're using Google collab, which we're using in open CME section, we're going to go 31 00:02:04,020 --> 00:02:06,690 back and do inter-club very soon, actually. 32 00:02:06,690 --> 00:02:12,660 And maybe the next three or four after the next three or four lectures, we'll start using CoLab to 33 00:02:12,660 --> 00:02:18,600 train with GPUs over convolutional neural networks, using both PyTorch Tensor for Keros. 34 00:02:19,170 --> 00:02:28,910 So you'll see how much faster these network streamer GPUs and these are you use to free up using CoLab. 35 00:02:29,790 --> 00:02:32,460 Honestly, they're kind of ancient, and they were quite slow. 36 00:02:32,820 --> 00:02:35,520 They are faster debuts be colab that you can pay for. 37 00:02:36,000 --> 00:02:37,590 I do pay for it because it's quite cheap. 38 00:02:37,590 --> 00:02:44,760 It's $10 a month and you get access to much faster P 100s or T100 GPUs if they're available. 39 00:02:45,330 --> 00:02:50,790 So it's actually a pretty decent substitute when you do have to do some simple models. 40 00:02:51,150 --> 00:02:53,610 So it's another thing as well. 41 00:02:53,730 --> 00:02:59,220 Deep learning and deep learning libraries have become much more mature, and they're quite easy to use, 42 00:02:59,220 --> 00:02:59,820 to be honest. 43 00:03:00,300 --> 00:03:04,110 It's very easy to train at CNN using cameras or by touch. 44 00:03:04,500 --> 00:03:06,630 It doesn't take that much knowledge to get up and running. 45 00:03:06,990 --> 00:03:10,440 However, you do need to understand if you need to make improvements or tweaks. 46 00:03:10,860 --> 00:03:15,810 You do need to understand the inner workings of deep learning, which you would have already achieve 47 00:03:15,990 --> 00:03:22,020 if you watched the previous hour of lectures and things weren't always this way. 48 00:03:22,320 --> 00:03:24,080 And let's talk a bit about that. 49 00:03:24,090 --> 00:03:25,500 Let's talk about how we got here. 50 00:03:26,130 --> 00:03:29,310 So let's talk about the history of in deep learning. 51 00:03:29,850 --> 00:03:33,930 So it has been around for a while, actually. 52 00:03:34,380 --> 00:03:38,670 I mean, a lot of science fiction writers obviously talked about machines and robots. 53 00:03:38,730 --> 00:03:41,940 I could think maybe like a hundred years ago or so. 54 00:03:42,450 --> 00:03:47,610 However, really, I research started like maybe after World War Two, even probably during World War 55 00:03:47,610 --> 00:03:55,950 Two, where a lot of these groundbreaking computer scientists and mathematicians were working on simple 56 00:03:55,950 --> 00:04:03,930 A.I. concepts like the first neuron for artificial neuron algorithm was in 1943 by these guys. 57 00:04:04,680 --> 00:04:07,020 So and then they continued a lot of work. 58 00:04:07,050 --> 00:04:12,090 They actually did some work where they actually use continuous, a continuous back propagation model 59 00:04:12,240 --> 00:04:15,690 based on the basics of it to train some of these these networks. 60 00:04:16,110 --> 00:04:18,510 So things were coming along, coming along nicely. 61 00:04:18,520 --> 00:04:26,670 However, back then, and even maybe till like five, 10, 10 years ago, we just simply didn't have 62 00:04:26,670 --> 00:04:31,350 the computing power and it was easily accessible to train these new neural networks. 63 00:04:31,890 --> 00:04:36,600 So a lot of alternative methods and machine learning were developed, just cycle modeling and machine 64 00:04:36,600 --> 00:04:36,930 learning. 65 00:04:38,130 --> 00:04:39,960 These were these were quite good, actually. 66 00:04:40,380 --> 00:04:44,790 However, they reached a point where they had diminishing returns, where we just couldn't get much 67 00:04:45,030 --> 00:04:46,440 better performance out of them. 68 00:04:47,160 --> 00:04:52,080 And oftentimes these algorithms were too slow and inefficient and also error prone. 69 00:04:52,500 --> 00:04:59,070 So they also suffered from the curse of dimensionality, which means that when your dataset grows. 70 00:04:59,570 --> 00:05:04,760 Just becomes exponentially harder to train that type of model using these methods. 71 00:05:06,740 --> 00:05:08,750 So the deep learning did change a lot of this. 72 00:05:10,070 --> 00:05:12,200 So let's talk about it a winter. 73 00:05:12,680 --> 00:05:20,720 Lots of people in the world don't even know what this is, but in the 1970s and in the late 60s, lots 74 00:05:20,720 --> 00:05:23,280 of people were very optimistic about it. 75 00:05:23,990 --> 00:05:31,040 They thought like computers that could understand video text, even talk back to us, even like some 76 00:05:31,040 --> 00:05:35,600 sort of semi consciousness was achievable within was achievable in their lifetime. 77 00:05:36,320 --> 00:05:41,540 However, as the as more research went along, it was it was. 78 00:05:41,570 --> 00:05:44,240 It was realized that this wasn't going to be possible. 79 00:05:44,720 --> 00:05:50,570 So a lot of funding was slashed and many cleaned air, which is hype, that it wasn't possible. 80 00:05:50,990 --> 00:05:54,800 So it kind of went into the depression, and this graph gives us a good example of it. 81 00:05:55,220 --> 00:06:01,670 So you can see this is the debate of genesis, of a lot of hype, a lot of speculation than a lot of 82 00:06:01,670 --> 00:06:07,280 failures and a lot of wasted money, which is important because a lot of this air work takes a lot of 83 00:06:07,280 --> 00:06:08,210 funding to achieve. 84 00:06:09,110 --> 00:06:10,310 So this was the winter. 85 00:06:10,910 --> 00:06:16,220 Then maybe in the 80s, when some machine learning algorithms started to make more progress and computers 86 00:06:16,220 --> 00:06:17,390 became more accessible. 87 00:06:17,780 --> 00:06:24,860 There was the early Commodore 64 and your old Apple Macintoshes, so a lot of people did think maybe 88 00:06:25,190 --> 00:06:27,500 we can probably start getting some simple UI stuff done. 89 00:06:28,040 --> 00:06:29,930 And then those hopes vanished again. 90 00:06:30,350 --> 00:06:31,370 And that was in the 90s. 91 00:06:32,090 --> 00:06:35,330 And maybe it's basically since then we've seen growth. 92 00:06:35,660 --> 00:06:40,010 This graph isn't really sure, surely explosive growth that we have seen in recent time, but we've 93 00:06:40,010 --> 00:06:41,290 seen growth in AI. 94 00:06:41,660 --> 00:06:43,060 It was incremental. 95 00:06:43,070 --> 00:06:48,920 It was it was lots of good algorithms, screams, machine learning algorithms came along. 96 00:06:49,370 --> 00:06:55,280 Then we started to do a simple convolutional neural networks and neural networks and elastic arms. 97 00:06:55,280 --> 00:06:59,330 And you don't know what all of these terms yet, but they're just types of neural networks. 98 00:06:59,810 --> 00:07:04,610 And it was it was the potential was seen and it grew and it grew and grew. 99 00:07:04,970 --> 00:07:11,360 And right now, honestly, right now, I think in the last year or two, I'm seeing so much improvement 100 00:07:11,360 --> 00:07:15,960 in computer vision world with all of this new models and methods we're using. 101 00:07:16,490 --> 00:07:20,570 It's been it's been an explosion, and I don't see it slowing down anytime soon because there's still 102 00:07:20,570 --> 00:07:26,480 so much work to be done in computer vision and in a simple, a simple thing like a robot. 103 00:07:26,480 --> 00:07:32,450 Navigating inside of a home requires so much computer vision, knowledge and processing, not just because 104 00:07:32,570 --> 00:07:37,470 he wants to navigate and avoid obstacles, but he has to understand his environment. 105 00:07:37,490 --> 00:07:39,530 Yes, understand that this is a door. 106 00:07:39,590 --> 00:07:40,520 This is a doorknob. 107 00:07:40,820 --> 00:07:41,740 This is a television. 108 00:07:41,750 --> 00:07:44,420 This is a sink or bathroom and you need needs. 109 00:07:44,600 --> 00:07:48,050 He needs to see and visually a process and understand the scene that he's in. 110 00:07:48,740 --> 00:07:50,570 So it's not that hard yet. 111 00:07:50,960 --> 00:07:56,930 But what's also hard is actually making this work in real time, processes information accurately and 112 00:07:56,930 --> 00:07:58,030 then make decisions out of it. 113 00:07:58,040 --> 00:08:04,390 So it's also tying into tying in together all of the the building blocks of an air system. 114 00:08:06,020 --> 00:08:13,580 So new cognition, cognition, sorry, was one of the first neural networks, and this was released 115 00:08:13,580 --> 00:08:14,510 in 1980. 116 00:08:15,110 --> 00:08:18,200 Actually, it was solid work in 1979 by these guys. 117 00:08:18,650 --> 00:08:23,080 And basically, this was like the grandfather of neural networks. 118 00:08:25,260 --> 00:08:30,810 And people actually did that propagation back in FORTRAN in the 70s, because back propagation is a 119 00:08:30,810 --> 00:08:32,490 relatively simple algorithm. 120 00:08:32,790 --> 00:08:37,860 Once you can do the numerical calculations to get the derivatives, it's relatively. 121 00:08:38,100 --> 00:08:39,600 I mean, I wouldn't say it's easy to implement. 122 00:08:39,750 --> 00:08:42,620 This is actually good looks, but it's probably a lot more code in this. 123 00:08:42,720 --> 00:08:45,150 It's probably just a small, tiny block of it. 124 00:08:45,810 --> 00:08:51,780 It's probably pretty complicated because I can imagine in Python using a lot of libraries as well. 125 00:08:51,990 --> 00:08:54,540 It's still relatively hard. 126 00:08:54,630 --> 00:09:00,150 I wouldn't say that difficult, but relatively hard to code a neural network from scratch and do the 127 00:09:00,150 --> 00:09:01,020 training process. 128 00:09:01,410 --> 00:09:07,470 However, it's a very, very good exercise if you want to understand deep learning and in reality. 129 00:09:08,010 --> 00:09:12,850 So if you do, you can search Google building a neural net from scratch in Python. 130 00:09:12,870 --> 00:09:14,640 There's a lot of blogs that talk about this. 131 00:09:14,670 --> 00:09:18,060 It's a very good exercise to do so. 132 00:09:18,810 --> 00:09:20,010 Things did pick up, though. 133 00:09:20,760 --> 00:09:26,460 As you can see, I talked about this here, the basically this point, but that was actually end of 134 00:09:26,460 --> 00:09:29,220 the year I went to when I started to create new hopes. 135 00:09:29,370 --> 00:09:31,350 So that just mentioning that there. 136 00:09:32,640 --> 00:09:34,110 So here's the thing. 137 00:09:34,740 --> 00:09:36,870 GPAs and deep learning are a match made in heaven. 138 00:09:37,320 --> 00:09:42,150 And yes, I know there are specialized deep learning hardware out there. 139 00:09:42,810 --> 00:09:46,980 However, no one is going to beat the ubiquity and accessibility of chips. 140 00:09:47,340 --> 00:09:51,450 Even though I know right now in 2021 to late 2021, I'm recording. 141 00:09:51,450 --> 00:09:53,880 This is still a GPU shortage. 142 00:09:54,420 --> 00:09:59,960 Like trying to get a 290 end video card is next to impossible sometimes. 143 00:10:01,080 --> 00:10:08,490 There was a company I used to do some work with that was trying to get this GPU since since April, 144 00:10:08,640 --> 00:10:11,940 that was about six months ago and still haven't gotten it yet. 145 00:10:14,820 --> 00:10:20,580 So think of a GP use as like a four door sedan, so sorry, CPU's as a four door sedan. 146 00:10:21,060 --> 00:10:26,970 They're very good all vehicles, but they're not good at specializing in one general thing. 147 00:10:27,450 --> 00:10:29,760 So that's why that's why we need GP use. 148 00:10:30,360 --> 00:10:35,130 But think about what GP use are and what gives them the capacity to be good at deep learning. 149 00:10:35,520 --> 00:10:36,930 Well, let's loosen that analogy. 150 00:10:37,470 --> 00:10:39,090 GP use are like a huge truck. 151 00:10:39,330 --> 00:10:41,670 They're good for wanting, but not good for other things. 152 00:10:42,030 --> 00:10:43,830 Handling is bad in trucks. 153 00:10:44,190 --> 00:10:45,720 It's not going to be a very comfortable ride. 154 00:10:46,320 --> 00:10:48,930 But at the same time, it can pull a lot of weight. 155 00:10:49,260 --> 00:10:57,420 So in essence, this is that's what the GP who is a GP you is does some calculations, some types of 156 00:10:57,420 --> 00:11:03,480 calculations, very, very efficiently, and those types of calculations can actually be skilled quite 157 00:11:03,480 --> 00:11:03,810 well. 158 00:11:03,990 --> 00:11:05,730 So he can use multiple cores. 159 00:11:06,090 --> 00:11:11,910 That's why in Nvidia's GPUs, they have like two thousand up to 4000 CUDA cores. 160 00:11:12,420 --> 00:11:18,180 And all of these things allow it to achieve parallel processing, which allows us to do so many calculations 161 00:11:18,180 --> 00:11:21,450 at once, and we can see a comparison afterwards. 162 00:11:21,480 --> 00:11:23,490 But for now, let's talk about that. 163 00:11:24,360 --> 00:11:31,230 While GPUs are mainly were originally used for treaty processing and I'm basically rendering scenes 164 00:11:31,230 --> 00:11:31,620 as well. 165 00:11:32,370 --> 00:11:36,420 They were actually not really designed for deep learning at that point. 166 00:11:36,780 --> 00:11:43,020 However, the reason they were used for deep learning processes is because they performed floating point 167 00:11:43,020 --> 00:11:47,040 operations very, very quickly and they have dedicated memory. 168 00:11:47,460 --> 00:11:53,150 So this allows high bandwidth movement of data within the GPU, so it becomes very, very good for this 169 00:11:53,330 --> 00:11:55,650 sort of task specific task. 170 00:11:57,390 --> 00:12:00,750 So you have come a long way because look at this here. 171 00:12:00,840 --> 00:12:08,760 In 2003, the best CPU just barely had but had more theoretical computation performance than a CPU. 172 00:12:09,150 --> 00:12:14,880 And that's understandable because GPUs do dedicated types of calculations and they're able to do it 173 00:12:14,880 --> 00:12:15,450 very well. 174 00:12:15,930 --> 00:12:21,780 However, look at the difference in computational performance GPUs made compared to CPUs. 175 00:12:22,080 --> 00:12:23,540 CPUs started around here. 176 00:12:23,550 --> 00:12:26,880 This is like maybe a two or three and right note. 177 00:12:26,940 --> 00:12:29,040 Even this is probably a lot more now. 178 00:12:29,130 --> 00:12:34,470 But even in the last five years, Djibouti CPUs have not improved that much. 179 00:12:34,980 --> 00:12:41,820 The last big jump in and CPU's with some of AMD's Threadripper CPUs. 180 00:12:42,180 --> 00:12:50,160 Also Apple's M1, which is a risk CPU for Abrams review, was outstanding compared to when you compared 181 00:12:50,160 --> 00:12:52,950 it to performance of its CPUs. 182 00:12:53,520 --> 00:12:57,240 But nevertheless is still pale in comparison to the actual theoretical. 183 00:12:57,240 --> 00:13:04,140 Computational performance of CPUs and GPUs have actually made so much strides in the last few years, 184 00:13:04,140 --> 00:13:07,980 like Nvidia's 3000 series is phenomenally fast. 185 00:13:08,670 --> 00:13:14,040 So I hope this lesson this lecture I know it wasn't a more smooth kind of well-put to get together lecture. 186 00:13:14,400 --> 00:13:19,560 However, I just wanted you to understand what's making deep learning so accessible right now. 187 00:13:19,860 --> 00:13:25,590 It's a combination of GPU use becoming cheaper and faster, deep learning libraries becoming much, 188 00:13:25,620 --> 00:13:28,140 much more mature and easy to use. 189 00:13:28,560 --> 00:13:34,920 And then all this research, all these research that's just bombarding us from all around because tech 190 00:13:35,220 --> 00:13:42,390 big big tech companies like Facebook, Google, Amazon, Netflix, Apple, Microsoft, they're they're 191 00:13:42,810 --> 00:13:48,450 pouring so much money and resources into deep learning, and they're open sourcing almost all of their 192 00:13:48,450 --> 00:13:51,120 software, at least all of the good software, in my opinion. 193 00:13:51,600 --> 00:13:57,300 So it's becoming very, very it's becoming a very good time to get into this field. 194 00:13:57,690 --> 00:14:04,380 So we'll stop there and we'll take a look now at some of the most common and popular deep learning libraries, 195 00:14:04,380 --> 00:14:06,070 which you will be using in discourse. 196 00:14:06,480 --> 00:14:08,040 So I'll see you in the next section. 197 00:14:08,310 --> 00:14:08,700 Thank you.