WEBVTT 00:00.150 --> 00:04.050 OK, let's do something a little more advanced with the prompter you will read and next logs and create 00:04.050 --> 00:10.830 a simple dashboard Engine X reverse proxy that was installed at the beginning of the course, he had 00:10.830 --> 00:14.700 the reverse proxy Bafana with Engine X. So every request is going by. 00:14.700 --> 00:20.170 That reverse proxy is being locked into a log file and make it read that using prompts in Loki. 00:20.530 --> 00:24.900 Also in Loki will use what's called the pattern parser, but will go into that. 00:24.900 --> 00:31.560 First, we open up our scraped config file conflict, prompt Obama on our Quanta server and add this 00:31.560 --> 00:32.620 extra section here. 00:32.640 --> 00:37.850 This is a second script config called Engine X. This target is local host. 00:37.860 --> 00:42.830 The job name is Engine X and the path is VAR Log Engine X Star Log. 00:42.950 --> 00:46.580 So going on to Mega-fauna Server, OK, someone might have found a server. 00:46.590 --> 00:49.010 We'll have a look at that folder where the logs are. 00:49.020 --> 00:54.090 So c d the logs and generics ls lh. 00:54.270 --> 01:00.120 There are the log files that Engine X is saving, and you can see that they're accessible via the IDM 01:00.120 --> 01:00.450 group. 01:00.570 --> 01:06.120 So our prompt, our user is already in the IDM group, but if you are using a specific user for prompt, 01:06.120 --> 01:09.450 they'll then make sure that user is in the IBM group so that it can read the logs. 01:09.580 --> 01:12.600 OK, so let's now edit the prompt her config file. 01:12.600 --> 01:19.450 So see the usr local bin Telis L'hygiene that was config prompt. 01:19.450 --> 01:24.510 Tell this so that the nano config from tail y eml. 01:24.520 --> 01:26.730 OK, this is my existing pronto. 01:26.760 --> 01:31.410 Remember, I've explicitly said that two nine nine seven you can let that to zero if you like. 01:31.420 --> 01:38.340 That's the URL that my local prompt tail is sending a sending to a local Loki as existing script config 01:38.340 --> 01:40.400 whose job name is bollocks. 01:40.410 --> 01:43.050 I added the host label there for gharana. 01:43.080 --> 01:49.170 Now I've positioned my cursor where I want to start pasting now going to just copy that part, including 01:49.170 --> 01:53.150 the whitespace control, see right click and a paste case. 01:53.160 --> 01:57.570 So job name and your next set of conflicts targets local host labels and generics. 01:57.570 --> 02:04.500 And that is the path to the log files that will read so VAR Log Engine X style log book save that control 02:04.500 --> 02:11.700 x yes and to its restart restart from tail and check its status. 02:12.060 --> 02:15.930 It looks good active running and I'm not seeing any problems. 02:16.350 --> 02:24.660 OK, so we can now go into Carafano and open up, explore and find a new entry here on the job code 02:24.660 --> 02:31.680 Engine X, so click Engine X and that is the log stream selector job equals Engine X and show logs, 02:32.120 --> 02:36.000 and we can begin to see logs that prompt tail is now pushing into Loki. 02:36.360 --> 02:43.190 OK, so we can see here the file name is VAR Log Engine X Access Log Host Crafar Nut Job Engine X. 02:43.200 --> 02:47.820 There's another one access local Engine X. And if we look at the details of the launch, we can see 02:47.820 --> 02:50.180 that was opposed to the Lokey service. 02:50.190 --> 02:53.970 That's the IP address of my Esquibel server using a low key push method. 02:54.030 --> 02:58.560 If remember, I set my prompt in the mosque will server to go via the Engine X reverse proxy where it 02:58.560 --> 02:59.940 was using the domain name. 02:59.940 --> 03:08.070 And so that IP address there is my actual server that I'm using to make this video and I'm making requests 03:08.070 --> 03:11.970 to the graphical user interface every time I press a button on the Griffon user interface. 03:12.000 --> 03:18.420 Anyway, there's a lot of information in these log lines here that we can query, but this is a good 03:18.420 --> 03:23.790 opportunity to learn a new feature in Loki, and that is the pattern parser. 03:24.330 --> 03:29.430 The pattern parser will allow us to take parts of those log lines and create labels from them. 03:29.460 --> 03:33.770 So, for example, job equals Engine X will pass a pattern over that. 03:33.780 --> 03:38.820 So pipe pattern we're matching a string and putting matches into labels. 03:38.820 --> 03:44.400 So in this pattern here, according to new labels called method and Status, if I look at the log line 03:44.400 --> 03:47.580 here, there's those two hyphens there, there's two hyphens there. 03:47.610 --> 03:53.550 So we could be taking the first property that IP address and putting that into a value or that one. 03:53.550 --> 03:56.120 But we're taking method and status. 03:56.130 --> 03:58.260 So those here, that's the method. 03:58.260 --> 04:01.590 The word post and status is the number two I four. 04:01.590 --> 04:02.810 So copy that line. 04:02.820 --> 04:05.340 Put that into your query to pop pattern. 04:05.340 --> 04:09.990 We're matching that pattern, that whole string and creating two new labels called method and Status. 04:09.990 --> 04:14.640 And if it can match that string and find values to put into method and status, it will create them 04:14.640 --> 04:15.720 as new labels for us. 04:15.800 --> 04:16.890 We'll say that's a shift. 04:16.890 --> 04:18.090 Enter now. 04:18.090 --> 04:23.610 If I look at one of these lines, I've now got two new labels here method and status, so I can now 04:23.610 --> 04:27.270 start using those labels further in my query here. 04:27.300 --> 04:30.810 So, for example, let's count all by time, so can credit graph. 04:31.260 --> 04:38.760 So going to the beginning count over time bracket, what to the end and we'll say for range of one minute 04:38.910 --> 04:41.460 and then we'll close that off with the bracket shift. 04:41.460 --> 04:44.910 And so we now start getting a graph of the zoom to that day. 04:44.940 --> 04:45.410 There we go. 04:45.420 --> 04:51.270 We can start to see the different kinds of methods and status codes that our engineers service proxy 04:51.270 --> 04:51.630 is things. 04:51.630 --> 04:55.770 So status 200 Forget method status two hundred post method. 04:55.800 --> 04:59.820 There was a status four hundred down there instead of two 04 if I zoom in. 04:59.900 --> 05:04.250 To say that section there that looks at the more interesting, these are the kinds of status codes that 05:04.250 --> 05:08.960 we're seeing and that quite common to means okay, but you might get lots of four or four errors and 05:08.960 --> 05:10.250 that means file not found. 05:10.280 --> 05:13.340 You might get sort of a five hundred errors, which means it's a problem with the application running 05:13.340 --> 05:15.370 behind your web server or reverse proxy. 05:15.380 --> 05:17.620 So I'll create a four for error now. 05:17.630 --> 05:24.350 So if I go to the recipe code dot net and it just type in some junk that will return for a four page 05:24.350 --> 05:25.070 not found for a. 05:25.550 --> 05:27.860 Now we'll see that you now end your next job now. 05:27.900 --> 05:33.800 OK, so if I change that query to the last five minutes, I just zoom into that section there. 05:34.310 --> 05:37.740 There is a for for just the red line there. 05:37.760 --> 05:40.760 So that's the four a that I just generated about 10 seconds ago. 05:40.790 --> 05:41.240 Excellent. 05:41.270 --> 05:45.980 So on a busy web server, it's good to see what all the status codes are because if you're suddenly 05:45.980 --> 05:48.760 getting status five hundreds, it will stand out like a sore thumb. 05:48.770 --> 05:52.210 If you see a sudden rise in four iPhones, you're not as a problem as well. 05:52.220 --> 05:55.550 And there are many status codes and you can look those up on the internet what they mean anyway. 05:55.580 --> 05:56.280 So that's good. 05:56.300 --> 05:59.530 So they're looking at the typical log lines that you get from an engineer. 05:59.630 --> 06:05.810 So it's a small sample here familiar on there are many values, so we can see this IP address requesting 06:05.810 --> 06:10.360 that's called remote address is a time that's called time local here. 06:10.370 --> 06:15.440 We don't have a remote user, but you might see that sometimes there is a method that's post. 06:15.440 --> 06:16.260 There is the quest. 06:16.260 --> 06:19.330 That's the path that was being requested from your web server. 06:19.340 --> 06:22.300 There was a protocol payslip one point one. 06:22.310 --> 06:26.790 You'll see different versions of how you stop being requested by move further along. 06:26.810 --> 06:31.820 That's the status code to have seen that Bytes sent zero HTP refer. 06:31.820 --> 06:32.910 We're not saying that day. 06:32.960 --> 06:34.130 You might find that value. 06:34.130 --> 06:38.720 Sometimes I use to be user agent pronto, but when I'm using my browser that I used to be user agent 06:38.720 --> 06:41.270 is usually something more complicated. 06:41.270 --> 06:47.360 Like that Mozilla five app, a web kit, etc. So all those values can be extracted by modifying our 06:47.360 --> 06:47.810 pattern. 06:47.840 --> 06:54.830 OK, so here's an example where I write labels for remote address and time like also copy that string 06:55.010 --> 06:56.780 and will replace the whole load. 06:57.110 --> 07:03.200 And so if I look at the labels, it now says remote address and time local soccer, my queries to refine 07:03.200 --> 07:04.820 on those two values if I needed to. 07:05.120 --> 07:11.240 What I'm going to do is modify this one and add method status back so that you can see that we can use 07:11.240 --> 07:12.500 all those values if we want. 07:12.600 --> 07:15.500 My address time or method was at that position. 07:15.500 --> 07:18.860 Their method and status was at that position. 07:18.860 --> 07:20.900 Their status shift. 07:20.900 --> 07:21.470 Enter. 07:21.520 --> 07:26.330 OK, if I look at the labels, I'm also sing status and method again as well. 07:26.340 --> 07:27.290 So method post. 07:27.320 --> 07:32.690 Now, it's not advisable to create variables for all of these things if you don't actually using them 07:32.690 --> 07:34.370 because it's just not good for performance. 07:34.380 --> 07:36.020 Pretty well, I'm just showing you that it's possible. 07:36.050 --> 07:41.270 Also, you can change the names of the labels anything you like if you prefer my address like that. 07:41.420 --> 07:44.630 For example, it now says from my address like that. 07:44.660 --> 07:45.890 So you got the freedom. 07:45.890 --> 07:49.840 So the Pattern Pass is actually really good and it's actually very fast as well. 07:49.850 --> 07:55.090 In the past, you would use something like rejects in that position, but they say the pattern parser 07:55.100 --> 07:56.660 is now the fastest way of doing this. 07:56.660 --> 07:58.250 So and it looks pretty easy as well. 07:58.250 --> 07:59.460 So we use a pattern passel. 07:59.510 --> 08:00.710 OK, so excellent. 08:00.740 --> 08:03.650 Now I'm just going to get rid of my address there, so I'm not going to use it. 08:03.660 --> 08:09.200 I'm not going to use time local either, but I'm going to create a graph from that, but also group 08:09.320 --> 08:09.850 as well. 08:09.860 --> 08:12.230 So because then I'll use that in a dashboard. 08:12.290 --> 08:20.090 So going back to count over time bracket for one minute and close off that bracket, I'm going to sum 08:20.090 --> 08:21.710 that some until. 08:21.710 --> 08:27.190 So we're creating one line and then I'm going to group by status there. 08:27.660 --> 08:31.120 So I've got a simple graph now that is just showing status codes. 08:31.130 --> 08:35.990 I don't really care about the method, but if I did care, I could just put in method like that and 08:35.990 --> 08:39.200 I've got the label standing using the method and the status. 08:39.290 --> 08:40.070 I'm not going to use it. 08:40.130 --> 08:43.040 Also, another thing that I haven't shown you as well. 08:43.040 --> 08:51.590 You can change the order of this grouping clause by saying some high status and then that's the remainder 08:51.590 --> 08:52.300 of the query. 08:52.480 --> 08:54.450 And so that's the same result. 08:54.500 --> 08:55.400 So that's an option. 08:55.400 --> 09:00.380 If you prefer written like that some by status, then your query that also works for method, some by 09:00.380 --> 09:04.790 status method, some by status method, job, if you wanted to. 09:04.790 --> 09:05.720 But I'll do that. 09:05.870 --> 09:07.370 Now I'm going to use that in a dashboard. 09:07.370 --> 09:14.690 So copy that and let's create a new dashboard that an empty panel select low-key pace that query and 09:14.690 --> 09:23.720 to apply and just save this very quickly, calling it Engine X save and we can just to start down to 09:23.720 --> 09:25.240 15 minutes, for example. 09:25.250 --> 09:25.770 And then we go. 09:25.820 --> 09:30.980 I can add a log panel as well, so I can see the related log files. 09:31.040 --> 09:36.740 So that's at a panel time series or chosen logs. 09:38.400 --> 09:39.060 Low key. 09:40.330 --> 09:41.350 Curly brackets. 09:42.060 --> 09:43.950 Job engineers. 09:45.120 --> 09:46.320 Very good apply. 09:47.450 --> 09:53.510 Position at that time is that for the last five minutes and I want to know something more about that, 09:53.510 --> 09:56.060 for example, I can zoom in or consuming. 09:56.180 --> 09:58.190 Excellent and see the related little clients. 09:58.340 --> 10:01.570 OK, so that's very quickly a basic Engine X dashboard. 10:01.610 --> 10:04.790 I'm going to pause the video, create something a little more complicated. 10:04.790 --> 10:10.820 So anyway, I have go on and buy an extra panel here, which uses the bar gauge there just to create 10:10.820 --> 10:17.370 a summary of the remote addresses and how many times they're making a call to my web server. 10:17.390 --> 10:18.260 That's the query there. 10:18.350 --> 10:19.670 I'm using the bar gauge. 10:19.670 --> 10:24.530 Some can have a time job Engine X pattern remote address for the time range. 10:24.540 --> 10:27.620 I'm using this dollar range instead of one minute I'm using. 10:28.770 --> 10:35.490 The range property there are remote address that means that when I change the time here, the numbers 10:35.610 --> 10:42.210 will be more reflective of how many times in that period the last five minutes we can see that one of 10:42.210 --> 10:47.640 these IP addresses is making a lot of requests to my server, so I could deny that IP address. 10:47.640 --> 10:50.760 If I wanted to put any way, I'll save that save. 10:51.000 --> 10:53.940 Let's go back to the dashboard and I'll just reposition it. 10:55.130 --> 11:02.630 Like so anyway, this dashboard, Jason, here I'll put on my documentation so they can copy and paste 11:02.810 --> 11:08.020 that is down here on the Sample and snakes dashboard, so you could copy that whole lot to the clipboard, 11:08.030 --> 11:13.310 go to dashboards, manage or save that for I go. 11:14.260 --> 11:21.610 Import by panel, Jason Paste that let me go to the copy load, that name already exists, so I'm going 11:21.610 --> 11:24.040 to change it to something else import. 11:24.250 --> 11:30.330 OK, so I've got that loaded so we can see here straight away what's going on with my next reverse proxy 11:30.340 --> 11:30.880 anyway. 11:31.360 --> 11:37.750 Just so that, you know, my Gravano server is under a current dose, so I'm getting a lot of junk actually 11:37.750 --> 11:38.920 being sent to the server. 11:38.920 --> 11:39.940 We can see that down here. 11:39.970 --> 11:43.870 So if your Gryphon, a server, is on the internet, there are possibilities you might start getting 11:43.880 --> 11:46.240 dosed if someone wants to detox you. 11:46.240 --> 11:48.080 So I'm using digital lotion. 11:48.080 --> 11:50.170 Digital Lotion has an ebook firewall. 11:50.500 --> 11:56.710 So, for example, on a networking under firewalls, you can create a firewall called anything you like. 11:56.830 --> 12:00.910 Set your inbound and outbound roles and you can apply it to a droplet. 12:00.940 --> 12:05.730 So, for example, I can apply it to my grandfather droplet, but I've already done that so far. 12:05.740 --> 12:09.020 Just go backwards and modify my existing one. 12:09.040 --> 12:15.850 Right now, I have all IP version four enabled for hACE2 TPS, so I'm just going to edit that rule. 12:15.880 --> 12:21.070 I'm going to delete that rule and just have those two explicit I-Pace that are allowed to query you 12:21.100 --> 12:24.040 to be a smug, often a server that's port four four three. 12:24.040 --> 12:25.810 So I'll save that now. 12:25.810 --> 12:32.040 If I go back into Jovana, we'll start to see that these numbers will start dropping down. 12:32.050 --> 12:34.090 So fast forward this video for a moment. 12:34.780 --> 12:40.690 So all these extra IP addresses on the right here are all being blocked, except for the two drive explicitly 12:40.690 --> 12:42.970 allowed in my firewall. 12:43.450 --> 12:48.340 One of those is my server that I'm creating this video from, and the other one is my MySchool. 12:48.340 --> 12:49.070 So OK. 12:49.090 --> 12:51.160 So you can see now that the graph is going down. 12:51.190 --> 12:55.010 So I go, that's one of the things that serves only to get divorced occasionally. 12:55.030 --> 12:55.510 Excellent. 12:56.680 --> 13:04.270 So if I looked at the last one minute see with this case, we can see that there are less remote addresses 13:04.270 --> 13:09.250 and eventually just be just the two, which are my two IP that I've explicitly allowed. 13:10.310 --> 13:10.760 Excellent.