1 00:00:12,620 --> 00:00:18,620 OK, so the next step is to set a random seed so that we get consistent results, feel free to comment 2 00:00:18,620 --> 00:00:20,960 this out in order to see different documents. 3 00:00:27,420 --> 00:00:32,850 The next step is to randomly choose a document and call the spin document function on that document. 4 00:00:39,220 --> 00:00:44,290 The next step is to print out our document using the text we're at module so that the text does not 5 00:00:44,290 --> 00:00:45,250 go off the screen. 6 00:00:50,730 --> 00:00:52,860 OK, so let's have a look at the results. 7 00:00:53,550 --> 00:00:58,530 Firstly, you'll notice that sometimes the original word is replaced with the same word. 8 00:00:59,160 --> 00:01:02,310 This is possible since the middle words are randomly selected. 9 00:01:02,670 --> 00:01:06,240 And of course, the original word is one of the possibilities. 10 00:01:07,020 --> 00:01:12,720 Note that if you wanted to force the word to change, you could temporarily set its probability to zero. 11 00:01:15,910 --> 00:01:19,930 OK, so notice that the first part of the first paragraph appears, OK. 12 00:01:20,530 --> 00:01:27,130 It says shares in train and plane making giant Bombardier have fallen to a 10 year low against the hands 13 00:01:27,130 --> 00:01:28,550 of its chief executives. 14 00:01:29,200 --> 00:01:34,660 It obviously doesn't make sense if you're a serious reader of this type of news, but it is grammatically 15 00:01:34,660 --> 00:01:35,290 correct. 16 00:01:35,860 --> 00:01:40,420 In fact, it's not difficult to find articles online that make even less sense. 17 00:01:41,970 --> 00:01:45,060 Note that the second part of the sentence does not make sense. 18 00:01:45,510 --> 00:01:48,450 It ends with in two members of the keyboard. 19 00:01:49,140 --> 00:01:51,330 So let's think about why this might happen. 20 00:01:52,140 --> 00:01:55,080 As you recall, a model looks at tri grams only. 21 00:01:55,140 --> 00:01:58,710 It does not consider the rest of the context in the sentence. 22 00:01:59,400 --> 00:02:03,870 So if you only look at this program, the keyboard, it makes complete sense. 23 00:02:04,380 --> 00:02:09,150 This is because you might have a sentence like the keyboard members were present at the meeting. 24 00:02:10,380 --> 00:02:16,080 But the reason this does not make sense is because the word of members appears earlier in the sentence. 25 00:02:16,650 --> 00:02:22,200 Thus, this is a hint that it might be useful to consider context beyond just the previous the next 26 00:02:22,200 --> 00:02:22,800 word. 27 00:02:25,390 --> 00:02:31,270 Let's now consider the next sentence, which says Paul Tellier, which was also Bombards epicenter, 28 00:02:31,600 --> 00:02:34,840 left the company amid an £80million restructuring. 29 00:02:36,680 --> 00:02:40,160 Again, it's kind of a weird sentence, but it's not completely wrong. 30 00:02:40,700 --> 00:02:44,060 It makes grammatical sense, but it does not make logical sense. 31 00:02:44,690 --> 00:02:50,270 So we normally do not use the word which to refer to a person, but rather who, which was the original 32 00:02:50,270 --> 00:02:50,810 word. 33 00:02:51,410 --> 00:02:56,690 Of course, our model doesn't know this because the previous token in the context was just a comma. 34 00:02:58,020 --> 00:03:03,990 Another funny replacement is Epicenter, which replaced the word president again, grammatically, it's 35 00:03:03,990 --> 00:03:10,110 not bad, but if you wanted to improve this, you would have to take into account very long range dependencies. 36 00:03:10,710 --> 00:03:15,870 Specifically, this word still refers to the first two tokens in the sentence, which are now quite 37 00:03:15,870 --> 00:03:17,280 a few tokens behind. 38 00:03:17,910 --> 00:03:23,430 So this suggests the need for a model that can perhaps look at a variable number of previous tokens 39 00:03:23,940 --> 00:03:26,970 and also one that can learn long range dependencies. 40 00:03:27,480 --> 00:03:31,260 Keep these ideas in mind when you study deep learning and neural networks. 41 00:03:33,990 --> 00:03:36,210 OK, so let's check the next sentence. 42 00:03:36,750 --> 00:03:42,900 Laurent Botwin, part of the family that controls the Montreal based firm, will take over the role 43 00:03:42,900 --> 00:03:46,290 of CEO under a newly created management structure. 44 00:03:47,100 --> 00:03:48,810 So this one makes perfect sense. 45 00:03:49,440 --> 00:03:54,380 By the way, notice how, for some reason, the de tokenize is failing at these points. 46 00:03:56,650 --> 00:03:58,660 So let's check out the next sentence. 47 00:03:59,170 --> 00:04:03,910 Analysts believe the resignations seem to have stemmed from a boardroom dispute. 48 00:04:04,570 --> 00:04:08,530 This also makes sense, but note that it changes the meaning of the sentence. 49 00:04:10,990 --> 00:04:17,769 OK, so the next sentence definitely does not make sense, although it is close under Mr Teles tenure 50 00:04:17,769 --> 00:04:25,480 at the subsidy, which began in July 2003, according to Cut, the world wide workforce of 75000 by 51 00:04:25,480 --> 00:04:28,780 signing a movement by two thousand six were announced. 52 00:04:29,500 --> 00:04:34,870 And of course, this does not make sense because the model does not account for long range dependencies. 53 00:04:38,300 --> 00:04:40,580 OK, so let's move on to the next sentence. 54 00:04:41,180 --> 00:04:47,870 The firm's auto division and Defense Services Unit were also sold in Bombardier started the future of 55 00:04:47,870 --> 00:04:52,700 a new aircraft seating 110 to 135 passengers. 56 00:04:53,240 --> 00:04:55,820 In this case, I would say that the result makes sense. 57 00:04:59,960 --> 00:05:02,840 So the next two sentences mostly make sense. 58 00:05:03,500 --> 00:05:10,670 Mr. Tellier had indicated he wanted to expand at the industry's top train maker and third largest manufacturer 59 00:05:10,670 --> 00:05:14,270 of civil aircraft until the restructuring was complete. 60 00:05:14,780 --> 00:05:18,980 But Bombardier has been charged with a declining share price and profits. 61 00:05:19,580 --> 00:05:22,130 In my opinion, these replacements are not that. 62 00:05:25,750 --> 00:05:29,110 So notice that the next sentence has quite a few replacements. 63 00:05:30,340 --> 00:05:34,620 Earlier this month, the government said it earned $100 million. 64 00:05:35,140 --> 00:05:42,880 Nineteen point two million pounds for the third quarter, down from a bid of $133 million a year ago. 65 00:05:43,630 --> 00:05:48,250 Again, this sentence doesn't make logical sense, but it does make grammatical sense. 66 00:05:51,420 --> 00:05:53,820 OK, so the next sentence is a bit of a mess. 67 00:05:54,420 --> 00:05:56,580 Notice that the first part looks OK. 68 00:05:57,240 --> 00:06:01,740 I understand the UK's concern that I would not be there for the long term. 69 00:06:02,490 --> 00:06:07,200 However, in the middle, we can see that the word end is replaced by a double quote. 70 00:06:07,860 --> 00:06:09,510 In the context of programs. 71 00:06:09,510 --> 00:06:14,760 This makes sense, but relative to the overall sentence, it does not make sense, since the quotes 72 00:06:14,770 --> 00:06:15,960 now don't balance out. 73 00:06:17,250 --> 00:06:20,130 Note that the ending of the sentence also looks OK. 74 00:06:20,760 --> 00:06:27,360 Mr Tellier said in a meeting on agriculture Obviously this doesn't make any sense or relative to the 75 00:06:27,360 --> 00:06:30,510 semantics of the article, but grammatically it works. 76 00:06:33,960 --> 00:06:38,040 OK, so since these articles are quite long, I think that's enough for this lecture. 77 00:06:38,730 --> 00:06:44,370 However, I do encourage you to look at more on your own and furthermore, think about why this article 78 00:06:44,370 --> 00:06:50,670 spinner behaves like it does in particular, think about things like the size of the context window 79 00:06:50,850 --> 00:06:52,480 and long term dependencies. 80 00:06:53,170 --> 00:06:58,920 It's also worth thinking about what level of understanding is really required to spin an article and 81 00:06:58,920 --> 00:07:00,780 make sure it makes sense semantically. 82 00:07:01,560 --> 00:07:04,320 That is, without changing the meaning of what it says. 83 00:07:04,980 --> 00:07:09,720 Consider whether or not anything like that currently exists at this time. 84 00:07:09,720 --> 00:07:15,720 The state of the art is GPT three, and the consensus is that we are close but still not there. 85 00:07:16,710 --> 00:07:22,350 Also consider whether or not anything like that is possible at all what level of intelligence would 86 00:07:22,350 --> 00:07:23,190 be required? 87 00:07:23,970 --> 00:07:28,830 From what I've seen, this is a much bigger leap than a lot of people think, especially for those who 88 00:07:28,830 --> 00:07:30,090 are new to machine learning. 89 00:07:31,810 --> 00:07:36,730 Inversion, one of this course, I was surprised at how many beginners thought something like human 90 00:07:36,730 --> 00:07:39,070 level understanding could just exist. 91 00:07:39,490 --> 00:07:42,940 And this was five years ago before Transformers were even invented. 92 00:07:43,690 --> 00:07:49,210 From my perspective, this just goes to show that there is a pretty wide gap between what I can do in 93 00:07:49,210 --> 00:07:52,570 reality versus what the public thinks I can do. 94 00:07:53,290 --> 00:07:58,150 So if you're new to this course or you're new to machine learning, it's a good idea to really think 95 00:07:58,150 --> 00:08:04,000 about whether your expectations are based on reality or if they are based on maybe misinformation that 96 00:08:04,000 --> 00:08:05,800 you got on YouTube or elsewhere. 97 00:08:10,390 --> 00:08:15,430 Now that being said, there are some ways you could extend this article spinner to make it better, 98 00:08:15,430 --> 00:08:18,610 even without considering deep learning and neural networks. 99 00:08:19,210 --> 00:08:22,780 Consider these to be extension exercises for this section. 100 00:08:24,100 --> 00:08:29,860 So as you recall, sometimes words that are replaced in a program makes sense in the context of the 101 00:08:29,860 --> 00:08:33,309 program, but not in the context of the full sentence. 102 00:08:33,970 --> 00:08:39,220 One way you could potentially address this issue is to take parts of speech into account. 103 00:08:39,820 --> 00:08:43,570 In this way, you'll at least have a greater chance of the result making sense. 104 00:08:45,180 --> 00:08:46,260 Here's another idea. 105 00:08:47,870 --> 00:08:51,680 Often we neglect to consider simple yet effective tools. 106 00:08:52,280 --> 00:08:56,270 This idea is to simply use a dictionary which contains synonyms. 107 00:08:57,320 --> 00:09:00,860 As you recall, a synonym means words that have the same meaning. 108 00:09:01,340 --> 00:09:04,430 For example, fast and quick are small and tiny. 109 00:09:05,240 --> 00:09:08,450 Note that this does not require any probabilistic model at all. 110 00:09:09,170 --> 00:09:13,490 Furthermore, you could also combine this approach with the parts of speech extension. 111 00:09:15,930 --> 00:09:21,420 Yet another option which has its pros and cons is to simply make the context window larger. 112 00:09:22,350 --> 00:09:26,790 For example, account for a more previous words and account for more future words. 113 00:09:27,570 --> 00:09:30,180 However, note that this introduces a data problem. 114 00:09:30,360 --> 00:09:36,360 Since you're increasing the dimensionality of the distribution, so this is another instance of the 115 00:09:36,360 --> 00:09:37,800 curse of dimensionality. 116 00:09:38,820 --> 00:09:43,650 Consider how unlikely you would be to encounter the same five gram more than once. 117 00:09:44,760 --> 00:09:49,760 Note also that the context window does not have to be symmetric, so you could depend on it. 118 00:09:49,770 --> 00:09:54,570 Two previous words, but only one next word or one previous words and two next words.