WEBVTT 00:00.440 --> 00:01.520 Hello everyone! 00:01.520 --> 00:06.440 In today's video we will cover detecting hallucination using model based approach. 00:06.840 --> 00:12.320 The model that we are going to learn and have hands on today is going to be vector based model which 00:12.320 --> 00:14.240 is hosted on hugging face. 00:14.280 --> 00:17.640 The model name is Hallucination Evaluation model. 00:17.640 --> 00:20.880 It's an open source model available on hugging face. 00:20.920 --> 00:23.840 Let's understand what this model has to offer. 00:23.840 --> 00:28.200 By hallucination we mean a text is not supported by other text. 00:28.560 --> 00:34.120 You always need two pieces of text to determine whether a text is hallucinated or not. 00:34.560 --> 00:41.240 When applied a rag, the LLM is provided with several pieces of text retrieved from some datasets, 00:41.600 --> 00:48.720 and a hallucination would indicate the summary or the hypothesis is not supported by those facts called 00:48.720 --> 00:49.520 evidence. 00:50.320 --> 00:54.560 A common type of hallucination in Iraq is factual but hallucinated. 00:54.800 --> 00:59.080 For example, given the premise the capital of France is Berlin. 00:59.560 --> 01:04.560 If the hypothesis is the capital of France is Paris is a hallucination. 01:04.560 --> 01:12.090 Although it is true in the real world scenario, this happens when do not generate based on the textual 01:12.090 --> 01:18.970 data provided to them as part of the Rag retrieval process, but rather generate content based on their 01:18.970 --> 01:20.490 pre-training knowledge. 01:21.010 --> 01:24.850 Let's run through a real world example in our Colab notebook. 01:25.210 --> 01:27.690 Okay, so here is my colab notebook. 01:28.170 --> 01:32.050 I have done a couple of predefined steps which take some time. 01:32.450 --> 01:34.050 I have installed the torch. 01:34.490 --> 01:41.690 I went ahead and logged into my huggingface account to download the model, and then I have also installed 01:41.690 --> 01:43.650 the FFT library. 01:44.130 --> 01:51.050 P is parameter efficient fine tuning of billion scale models on low resource hardware. 01:51.450 --> 01:57.130 This library would help us run model efficiently and optimize it on a low resource hardware. 01:57.490 --> 02:00.050 So that's the steps that I have already taken. 02:00.490 --> 02:02.850 Now let me go ahead and download the model. 02:03.490 --> 02:05.770 For that I am going to use auto model. 02:06.690 --> 02:10.610 Sequence classification is a transformers utility. 02:10.970 --> 02:14.770 And from there I'm going to download the vector hallucination model. 02:15.050 --> 02:16.050 Let me run this. 02:16.530 --> 02:19.290 It's going to take a while for it to download the model. 02:19.810 --> 02:21.130 Let me pause the video. 02:21.730 --> 02:23.690 So it downloaded the model. 02:24.010 --> 02:27.130 Now let me import the pipeline and tokenizer. 02:27.970 --> 02:28.850 All right. 02:28.850 --> 02:31.050 So I have imported the tokenizers. 02:31.250 --> 02:34.450 Now what I'm going to do is I'm going to provide two things. 02:34.690 --> 02:36.850 One is the test data. 02:37.090 --> 02:38.490 So that let's start with that. 02:38.770 --> 02:40.210 So this is the test data. 02:40.210 --> 02:42.330 So the capital of France is Berlin. 02:42.570 --> 02:44.690 The capital of France is Paris. 02:45.690 --> 02:47.650 So there are two parts to this. 02:47.890 --> 02:51.570 One is the premise, which is a fact that we are giving to the model. 02:51.770 --> 02:55.970 And now we know that the capital of France is Paris, which is right. 02:56.330 --> 03:03.130 But this is the, the factual data that we are giving to the to the model that consider the out the 03:03.170 --> 03:05.450 capital of France as Berlin. 03:05.690 --> 03:09.090 But this is what the response we got from the from the model. 03:10.010 --> 03:13.170 It's very important to understand these two things. 03:13.530 --> 03:13.930 Here. 03:13.930 --> 03:17.530 This is the fact that we are providing to the to the model. 03:17.850 --> 03:20.610 But this is what it it responds back with. 03:21.010 --> 03:23.050 So this is what it learned from the training. 03:23.050 --> 03:28.700 While this is what is being told as the context that we give to the to the model. 03:29.260 --> 03:31.700 Now, once we do that, let's provide the prompt here. 03:32.900 --> 03:36.700 So determine if the hypothesis is true given the premise. 03:36.700 --> 03:39.820 And the premise is a text one which is this. 03:40.260 --> 03:45.540 This is the premise and this is the hypothesis that we got from the the model response. 03:45.940 --> 03:49.060 Now let's run through the the loop of the list. 03:49.300 --> 03:53.700 So we will format the prompt pair zero is this value. 03:54.140 --> 03:58.140 It goes here in the premise whereas x2 is hypothesis. 03:59.220 --> 04:01.260 So now let's go ahead and create the pipeline. 04:02.140 --> 04:05.580 I'm going to do here is this classifier as the model. 04:05.940 --> 04:09.820 Go ahead and do a pipeline which is a text classification. 04:10.100 --> 04:13.940 We go to model as we have the model name which is vector based. 04:14.700 --> 04:15.260 Okay. 04:15.660 --> 04:18.300 And then the last part that we have here is tokenizer. 04:18.540 --> 04:21.900 We want to use auto tokenizer from pre-trained. 04:22.100 --> 04:26.660 I'm going to use the Google T5 base tokenizer. 04:26.980 --> 04:28.620 And that should be good enough. 04:28.900 --> 04:32.020 And then let's get the full scores from the classifier. 04:32.020 --> 04:34.820 Input is top k equals none. 04:35.180 --> 04:36.940 And then let's print this course. 04:37.140 --> 04:42.820 So what we're doing is we are executing the pipeline classifiers on it, and then we are running the 04:42.820 --> 04:43.900 input pairs. 04:44.100 --> 04:47.020 Top case none and printing the full scores. 04:47.260 --> 04:48.700 Let's go ahead and run this. 04:48.780 --> 04:49.340 All right. 04:49.940 --> 04:51.180 It's executing. 04:51.620 --> 04:52.740 There is a problem. 04:52.740 --> 04:54.780 Let's see what the what the problem is. 04:54.820 --> 04:56.380 There is a pipeline error. 04:56.660 --> 05:00.140 I had to add the part that says trust remote code as true. 05:01.260 --> 05:03.180 Now let's run this one more time. 05:03.420 --> 05:05.220 We are still having a problem. 05:05.660 --> 05:07.540 Unknown task classification. 05:08.700 --> 05:09.860 I'm just type the name. 05:10.940 --> 05:12.580 Now let's execute this. 05:13.860 --> 05:17.140 So we got an answer here, which is how much the model hallucinated. 05:17.460 --> 05:22.660 So if you notice here, for each premise we saw that this is the response we got. 05:23.060 --> 05:25.660 The the capital of France is Berlin. 05:25.860 --> 05:28.500 That's the response we got from the model. 05:28.740 --> 05:32.500 And the capital of France is Paris is actually the the right answer. 05:33.340 --> 05:40.860 So if you see here this it categorized this as hallucination whereas this was consistent whereas this 05:40.860 --> 05:42.180 was hallucination. 05:42.220 --> 05:42.780 All right. 05:43.340 --> 05:46.020 Thank you I will see you in the next video.